WASDI Configuration Documentation ================================= This document describes the structure and configuration options for the ``wasdiConfig.json`` file, which contains all WASDI system configuration settings. Overview -------- The main WASDI configuration is stored in a single JSON file that represents the ``WasdiConfig`` Java object and all its child configuration objects. This configuration is used by all WASDI components including: - Launcher - Trigger - Scheduler - Web Server - Database Utils Main ------ These are the configuration properties at root level of the JSON file. :: ### Root Level Properties .. code:: json { "nodeCode": "wasdi", "printServerAddress": "", "mainNodeCloud": "CREODIAS", "usersDefaultNode": "wasdi", "defaultSkin": "wasdi", "activateSubscriptionChecks": true, "systemUserName": "appwasdi", "systemUserId": 2042, "systemGroupName": "appwasdi", "systemGroupId": 2042, "baseUrl": "https://www.wasdi.net/wasdiwebserver/rest/", "connectionTimeout": 10000, "readTimeout": 10000, "msWaitAfterChmod": 1000, "shellExecLocally": true, "useLog4J": true, "logHttpCalls": true, "logLevel": "INFO", "logLevelServer": "", "logLevelLauncher": "", "logLevelScheduler": "", "logLevelTrigger": "", "addDateTimeToLogs": false, "filterInternalHttpCalls": true, "useNotebooksDockerAddress": false, "nvidiaGPUAvailable": false } Root Properties Description ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------+-------------+-------------------------------+ | Property | Type | Description | +===============================+=============+===============================+ | ``nodeCode`` | String | Unique identifier for this | | | | WASDI node (default: “wasdi”) | +-------------------------------+-------------+-------------------------------+ | ``printServerAddress`` | String | URL of the WASDI print server | +-------------------------------+-------------+-------------------------------+ | ``mainNodeCloud`` | String | Cloud provider of the main | | | | node (e.g., “CREODIAS”). | | | | Cloud Providers are just | | | | strings. They are used here | | | | in the main section, and in | | | | the dataProviders Config. | | | | This cloud name is used to | | | | understand if the node is in | | | | the same cloud of the | | | | dataProvider or not, and this | | | | information is useful to | | | | select the best data fetching | | | | option. | +-------------------------------+-------------+-------------------------------+ | ``usersDefaultNode`` | String | Default node assigned to new | | | | users | +-------------------------------+-------------+-------------------------------+ | ``defaultSkin`` | String | Default UI skin to apply | | | | (default: “wasdi”) | +-------------------------------+-------------+-------------------------------+ | ``activateSubscriptionChecks``| Boolean | Flag to activate or not the | | | | check and management of | | | | subscriptions | +-------------------------------+-------------+-------------------------------+ | ``systemUserName`` | String | System user name for WASDI | | | | processes (default: | | | | “appwasdi”) | +-------------------------------+-------------+-------------------------------+ | ``systemUserId`` | Integer | System user ID (default: | | | | 2042) | +-------------------------------+-------------+-------------------------------+ | ``systemGroupName`` | String | System group name (default: | | | | “appwasdi”) | +-------------------------------+-------------+-------------------------------+ | ``systemGroupId`` | Integer | System group ID (default: | | | | 2042) | +-------------------------------+-------------+-------------------------------+ | ``baseUrl`` | String | Base URL of WASDI API | +-------------------------------+-------------+-------------------------------+ | ``connectionTimeout`` | Integer | Connection timeout for | | | | third-party API calls (ms) | +-------------------------------+-------------+-------------------------------+ | ``readTimeout`` | Integer | Read timeout for third-party | | | | API calls (ms) | +-------------------------------+-------------+-------------------------------+ | ``msWaitAfterChmod`` | Integer | Wait time after chmod | | | | commands (ms) | +-------------------------------+-------------+-------------------------------+ | ``shellExecLocally`` | Boolean | Execute external components | | | | locally vs using Docker | | | | commands. WASDI uses | | | | different command line tools | | | | (e.g., gdalinfo, gdal_warp) | | | | and Python scripts for | | | | utilities and data providers. | | | | When ``true``, commands are | | | | executed as system-based | | | | shell exec (fully tested on | | | | Ubuntu, mostly compatible | | | | with Windows). When | | | | ``false``, WASDI uses fully | | | | dockerized mode and checks | | | | each shell exec command | | | | against the | | | | ``shellExecCommands`` | | | | configuration to determine | | | | whether to use system shell | | | | exec or docker run command. | | | | See Docker Configuration | | | | section for more details. | +-------------------------------+-------------+-------------------------------+ | ``useLog4J`` | Boolean | Use Log4J configuration vs | | | | standard output | +-------------------------------+-------------+-------------------------------+ | ``logHttpCalls`` | Boolean | Enable HTTP call logging | +-------------------------------+-------------+-------------------------------+ | ``logLevel`` | String | General log level | +-------------------------------+-------------+-------------------------------+ | ``logLevelServer`` | String | Web server specific log level | +-------------------------------+-------------+-------------------------------+ | ``logLevelLauncher`` | String | Launcher specific log level | +-------------------------------+-------------+-------------------------------+ | ``logLevelScheduler`` | String | Scheduler specific log level | +-------------------------------+-------------+-------------------------------+ | ``logLevelTrigger`` | String | Trigger specific log level | +-------------------------------+-------------+-------------------------------+ | ``addDateTimeToLogs`` | Boolean | Add timestamp to log lines | +-------------------------------+-------------+-------------------------------+ | ``filterInternalHttpCalls`` | Boolean | Filter internal HTTP calls | | | | from HTTP logs | +-------------------------------+-------------+-------------------------------+ | ``useNotebooksDockerAddress`` | Boolean | Use Docker internal names for | | | | Jupyter notebooks | +-------------------------------+-------------+-------------------------------+ | ``nvidiaGPUAvailable`` | Boolean | Enable NVIDIA GPU support for | | | | containers: set it true if | | | | this node have a GPU | +-------------------------------+-------------+-------------------------------+ Database Configuration ---------------------- MongoDB Configurations ^^^^^^^^^^^^^^^^^^^^^^^^^^^ WASDI supports multiple MongoDB instances for different purposes: .. code:: json { "mongoMain": { "address": "localhost", "port": 27017, "dbName": "wasdi", "user": "username", "password": "password", "replicaName": "" }, "mongoLocal": { "address": "localhost", "port": 27017, "dbName": "wasdilocal", "user": "username", "password": "password", "replicaName": "" } } The most important is the mongoMain section: this must target the main Mongo Db of the main node. All the computational nodes MUST be able to reach this database: usually this is done using an SSH tunnel. The main node MUST have only mongoMain. Any computational node MUST have mongoLocal: this is a small subset of the main database with only the collections that are used locally by the computational node. MongoDB Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ =============== ======= ================================ Property Type Description =============== ======= ================================ ``address`` String MongoDB server address ``port`` Integer MongoDB server port ``dbName`` String Database name ``user`` String Database username ``password`` String Database password ``replicaName`` String Replica set name (if applicable) =============== ======= ================================ Authentication Configuration ---------------------------- Settings about the authentication system. Keycloak Configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Keycloak is used as authentication server for WASDI. .. code:: json { "keycloack": { "address": "https://auth.wasdi.net", "cliSecret": "cli-secret", "authTokenAddress": "https://auth.wasdi.net/auth/realms/wasdi/protocol/openid-connect/token", "introspectAddress": "https://auth.wasdi.net/auth/realms/wasdi/protocol/openid-connect/token/introspect", "confidentialClient": "wasdi-confidential", "client": "wasdi", "clientSecret": "client-secret", "realm": "wasdi", "sessionExpireHours": 24 } } Keycloak Properties ^^^^^^^^^^^^^^^^^^^ ====================== ======= ================================ Property Type Description ====================== ======= ================================ ``address`` String Keycloak server address ``cliSecret`` String CLI client secret ``authTokenAddress`` String Token endpoint URL ``introspectAddress`` String Token introspection endpoint URL ``confidentialClient`` String Confidential client name ``client`` String Public client name ``clientSecret`` String Client secret ``realm`` String Keycloak realm name ``sessionExpireHours`` Integer Session expiration time in hours ====================== ======= ================================ File System Paths Configuration ------------------------------- In this section all the main paths needed by WASDI are configured. .. code:: json { "paths": { "downloadRootPath": "/data/wasdi/", "serializationPath": "/data/wasdi/params/", "metadataPath": "/data/wasdi/metadata/", "dockerTemplatePath": "/data/wasdi/dockertemplate/", "sftpRootPath": "/data/wasdi/sftp/", "geoserverDataDir": "/data/wasdi/geoserver/", "sen2CorePath": "/usr/local/sen2cor/", "userHomePath": "/home/appwasdi/", "missionsConfigFilePath": "/opt/wasdi/config/missions.json", "gdalPath": "/usr/bin/", "wasdiTempFolder": "/tmp/wasdi/", "pythonExecPath": "/usr/bin/python3", "traefikMountedVolume": "/data/wasdi/traefik/", "s3VolumesBasePath": "/mnt/wasdi/users-volumes/", "wasdiConfigFilePath": "/opt/wasdi/config/wasdiConfig.json" } } Paths Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------------------------+-------------+-------------------------------+ | Property | Type | Description | +===========================+=============+===============================+ | ``downloadRootPath`` | String | Base root path containing | | | | subfolders (workspaces, | | | | metadata, styles, workflows, | | | | processors, images) | +---------------------------+-------------+-------------------------------+ | ``serializationPath`` | String | Path where parameters are | | | | serialized | +---------------------------+-------------+-------------------------------+ | ``metadataPath`` | String | Metadata files path | +---------------------------+-------------+-------------------------------+ | ``dockerTemplatePath`` | String | Docker templates path | | | | | +---------------------------+-------------+-------------------------------+ | ``sftpRootPath`` | String | Root of the local SFTP server | +---------------------------+-------------+-------------------------------+ | ``geoserverDataDir`` | String | Geoserver data directory | +---------------------------+-------------+-------------------------------+ | ``sen2CorePath`` | String | Sen2Core binary path | +---------------------------+-------------+-------------------------------+ | ``userHomePath`` | String | User home path | +---------------------------+-------------+-------------------------------+ | ``missionsConfigFilePath``| String | Missions configuration file | | | | path | +---------------------------+-------------+-------------------------------+ | ``gdalPath`` | String | GDAL binary path | +---------------------------+-------------+-------------------------------+ | ``wasdiTempFolder`` | String | Temporary files folder | +---------------------------+-------------+-------------------------------+ | ``pythonExecPath`` | String | Full path to Python | | | | executable | +---------------------------+-------------+-------------------------------+ | ``traefikMountedVolume`` | String | Traefik mounted volume path | +---------------------------+-------------+-------------------------------+ | ``s3VolumesBasePath`` | String | S3 volumes mount folder path | +---------------------------+-------------+-------------------------------+ |``wasdiConfigFilePath`` | String | Path to this configuration | | | | file | +---------------------------+-------------+-------------------------------+ Docker Configuration -------------------- .. code:: json { "dockers": { "extraHosts": [], "pipInstallWasdiAddress": "https://pypi.org/simple/", "internalDockersBaseAddress": "localhost", "numberOfAttemptsToPingTheServer": 4, "millisBetweenAttmpts": 5000, "millisBetweenStatusPolling": 1000, "numberOfPollStatusPollingCycleForLog": 30, "millisWaitAfterDelete": 15000, "millisWaitAfterDeployScriptCreated": 2000, "millisWaitForLogin": 4000, "dockerComposeCommand": "docker-compose", "internalDockerAPIAddress": "http://127.0.0.1:2375/", "dockersDeployLogFilePath": "/var/log/wasdi/dockers.log", "logDockerAPICallsPayload": false, "removeDockersAfterShellExec": true, "removeParameterFilesForPythonsShellExec": true, "dockerNetworkMode": "net-wasdi", "processorsInternalPort": 5000, "groupAdd": [], "registers": [], "shellExecCommands": {}, "processorTypes": [], "eoepca": {} } } Docker Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------------------------------------------+---------------+-------------------------------+ | Property | Type | Description | +=============================================+===============+===============================+ | ``extraHosts`` | Array[String] | Extra hosts to add to | | | | containers (may be needed in | | | | some clouds for network | | | | reasons) | +---------------------------------------------+---------------+-------------------------------+ |``pipInstallWasdiAddress`` | String | Address to use to access PyPI | | | | to install waspy | +---------------------------------------------+---------------+-------------------------------+ | ``internalDockersBaseAddress`` | String | Address to use to reach | | | | internal dockers (default: | | | | “localhost”) | +---------------------------------------------+---------------+-------------------------------+ |``numberOfAttemptsToPingTheServer`` | Integer | Number of attempts to ping | | | | server before deciding it’s | | | | down (default: 4) | +---------------------------------------------+---------------+-------------------------------+ | ``millisBetweenAttmpts`` | Integer | Time (ms) to wait between | | | | attempts to check if docker | | | | is started (default: 5000) | +---------------------------------------------+---------------+-------------------------------+ | ``millisBetweenStatusPolling`` | Integer | Time (ms) to wait between | | | | calls to docker engine API | | | | for status (default: 1000) | +---------------------------------------------+---------------+-------------------------------+ | ``numberOfPollStatusPollingCycleForLog`` | Integer | Number of cycles before | | | | waitContainer function logs | | | | activity (default: 30) | +---------------------------------------------+---------------+-------------------------------+ | ``millisWaitAfterDelete`` | Integer | Time (ms) to wait for docker | | | | to complete delete operation | | | | (default: 15000) | +---------------------------------------------+---------------+-------------------------------+ | ``millisWaitAfterDeployScriptCreated`` | Integer | Time (ms) to wait after | | | | deploy.sh file creation | | | | (default: 2000) | +---------------------------------------------+---------------+-------------------------------+ | ``millisWaitForLogin`` | Integer | Time (ms) to wait for docker | | | | login operation (default: | | | | 4000) | +---------------------------------------------+---------------+-------------------------------+ | ``dockerComposeCommand`` | String | Command to use to start | | | | docker compose (default: | | | | “docker-compose”) | +---------------------------------------------+---------------+-------------------------------+ | ``internalDockerAPIAddress`` | String | Address of the local Docker | | | | instance API (default: | | | | “http://127.0.0.1:2375/”) | +---------------------------------------------+---------------+-------------------------------+ | ``dockerDeployLogFilePath`` | String | Path of file with docker | | | | build logs (default: | | | | “/var/log/wasdi/dockers.log”) | +---------------------------------------------+---------------+-------------------------------+ | ``logDockerAPICallsPayload`` | Boolean | Enable logging of payload for | | | | Docker Engine API calls | | | | (default: false) | +---------------------------------------------+---------------+-------------------------------+ | ``removeDockerContainersAfterShellExec`` | Boolean | Remove containers after shell | | | | execute (true for production, | | | | false for debug) | +---------------------------------------------+---------------+-------------------------------+ | ``removeParameterFilesForPythonsShellExec`` | Boolean | Remove input/output files | | | | after Python shell exec (true | | | | for production) | +---------------------------------------------+---------------+-------------------------------+ | ``dockerNetworkMode`` | String | Docker network mode (default: | | | | “net-wasdi”, can be | | | | overridden by shell exec | | | | items) | +---------------------------------------------+---------------+-------------------------------+ | ``processorsInternalPort`` | Integer | Standard processors internal | | | | port (default: 5000) | +---------------------------------------------+---------------+-------------------------------+ | ``groupAdd`` | Array[String] | List of group IDs to add to | | | | docker create command | +---------------------------------------------+---------------+-------------------------------+ | ``registers`` | Array | List of supported docker | | | [DockerRegi- | registries | | | stryConfig] | | +---------------------------------------------+---------------+-------------------------------+ | ``shellExecCommands`` | Map[String, | Map local shell exec commands | | | ShellExec | to equivalent docker | | | ItemConfig] | commands. When | | | | ``shellExecLocally`` is | | | | false, WASDI checks each | | | | shell exec command against | | | | this configuration. If a | | | | command (e.g., “gdalinfo”, | | | | “wasdi-launcher”) is found in | | | | this map, WASDI executes a | | | | docker run command instead of | | | | the classic shell exec. This | | | | enables fully dockerized | | | | WASDI deployment where all | | | | external tools run in | | | | containers. | +---------------------------------------------+---------------+-------------------------------+ | ``processorTypes`` | Array[Proces- | Configuration of processor | | | sorTypeConfig]| types | +---------------------------------------------+---------------+-------------------------------+ | ``eoepca`` | EOEPCAConfig | Configuration of EoEpca | | | | related docker parameters | +---------------------------------------------+---------------+-------------------------------+ ShellExecItemConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Each ``ShellExecItemConfig`` object in the ``shellExecCommands`` map contains the following properties: +---------------------------+-------------+-------------------------------+ | Property | Type | Description | +===========================+=============+===============================+ | ``dockerImage`` | String | Name of the docker image to | | | | use instead of the command | +---------------------------+-------------+-------------------------------+ | ``containerVersion`` | String | Version of the docker image | | | | to use | +---------------------------+-------------+-------------------------------+ | ``includeFirstCommand`` | Boolean | If true (default), all | | | | command line parts are passed | | | | as docker args. If false, the | | | | first element is not passed | | | | to docker command line | +---------------------------+-------------+-------------------------------+ | ``forceLocal`` | Boolean | If true, the command is | | | | executed locally even if | | | | WASDI is configured to be | | | | dockerized (default: false) | +---------------------------+-------------+-------------------------------+ | ``removePathFromFirstArg``| Boolean | If true, WASDI removes the | | | | path from the first command | | | | in the arg list | +---------------------------+-------------+-------------------------------+ | ``addPrefixToCommand`` | String | Prefix that will be added as | | | | arg[0] of the shell execute | +---------------------------+-------------+-------------------------------+ | ``additionalMountPoints`` |Array[String]| List of additional mount | | | | points for this specific | | | | docker | +---------------------------+-------------+-------------------------------+ | ``overrideDockerConfig`` | Boolean | Enable override of | | | | system/docker config (user, | | | | group, network) with specific | | | | values for this docker | +---------------------------+-------------+-------------------------------+ | ``systemUserName`` | String | System user name for this | | | | docker (default: “appwasdi”) | +---------------------------+-------------+-------------------------------+ | ``systemUserId`` | Integer | System user ID for this | | | | docker (default: 2042) | +---------------------------+-------------+-------------------------------+ | ``systemGroupName`` | String | System group name for this | | | | docker (default: “appwasdi”) | +---------------------------+-------------+-------------------------------+ | ``systemGroupId`` | Integer | System group ID for this | | | | docker (default: 2042) | +---------------------------+-------------+-------------------------------+ | ``dockerNetworkMode`` | String | Docker network mode for this | | | | container (default: | | | | “net-wasdi”) | +---------------------------+-------------+-------------------------------+ ProcessorTypeConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Each ``ProcessorTypeConfig`` object in the ``processorTypes`` array contains the following properties: +----------------------------------------+---------------+-------------------------------+ | Property | Type | Description | +========================================+===============+===============================+ | ``processorType`` | String | Name of the processor type | +----------------------------------------+---------------+-------------------------------+ | ``additionalMountPoints`` | Array[String] | List of additional mount | | | | points for this specific | | | | docker | +----------------------------------------+---------------+-------------------------------+ | ``commands`` | Array[String] | List of additional commands | | | | for this specific docker | +----------------------------------------+---------------+-------------------------------+ | ``environmentVariables`` | Array[Enviro- | List of environment variables | | | nmentVariable | to pass when creating the | | | Config] | container | +----------------------------------------+---------------+-------------------------------+ | ``image`` | String | Name of the base image to use | | | | (if needed) | +----------------------------------------+---------------+-------------------------------+ | ``version`` | String | Version of the base image to | | | | use (if needed) | +----------------------------------------+---------------+-------------------------------+ | ``extraHosts`` | Array[String] | Personalized extra hosts for | | | | this processor type | +----------------------------------------+---------------+-------------------------------+ | ``mountOnlyWorkspaceFolder`` | Boolean | If true, mount only | | | | ``/data/wasdi/[usr]/[wsid]/`` | | | | folder instead of entire | | | | ``/data/wasdi`` folder | +----------------------------------------+---------------+-------------------------------+ | ``templateFilesToExcludeFromDownload`` | Array[String] | List of file names that must | | | | not be downloaded when | | | | zipping the processor | +----------------------------------------+---------------+-------------------------------+ Docker Registry Configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code:: json { "registers": [ { "address": "nexus.wasdi.net", "user": "username", "password": "password", "priority": 1, "isDefault": true } ] } Docker Registry Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-----------------------+-------------+-------------------------------+ | Property | Type | Description | +=======================+=============+===============================+ | ``id`` | String | Unique identifier of the | | | | registry | +-----------------------+-------------+-------------------------------+ | ``user`` | String | Username for registry | | | | authentication | +-----------------------+-------------+-------------------------------+ | ``password`` | String | Password for the above user | +-----------------------+-------------+-------------------------------+ | ``address`` | String | HTTP address of the registry | +-----------------------+-------------+-------------------------------+ | ``priority`` | Integer | Priority of the registry | | | | (lower numbers = higher | | | | priority) | +-----------------------+-------------+-------------------------------+ | ``apiAddress`` | String | HTTP address of the registry | | | | API | +-----------------------+-------------+-------------------------------+ | ``repositoryName`` | String | Name of the repository inside | | | | the registry (default: | | | | “wasdi-docker”) | +-----------------------+-------------+-------------------------------+ Messaging Configuration ----------------------- Configurations related to messages. RabbitMQ Configuration ^^^^^^^^^^^^^^^^^^^^^^ .. code:: json { "rabbit": { "host": "localhost", "user": "guest", "password": "guest", "port": 5672, "queueName": "wasdi" } } RabbitMQ Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ============ ======= ========================================== Property Type Description ============ ======= ========================================== ``user`` String RabbitMQ username for authentication ``password`` String RabbitMQ password for the above user ``host`` String RabbitMQ server hostname or IP address ``port`` Integer RabbitMQ server port (default: 5672) ``exchange`` String RabbitMQ exchange name for message routing ============ ======= ========================================== Notifications Configuration --------------------------- .. code:: json { "notifications": { "mercuriusAPIAddress": "https://mercurius.cimafoundation.org/", "pwRecoveryMailTitle": "WASDI Password Recovery", "pwRecoveryMailSender": "noreply@wasdi.net", "pwRecoveryMailText": "Password recovery email text", "sftpMailTitle": "WASDI SFTP Account", "sftpManagementMailSender": "noreply@wasdi.net", "sftpMailText": "SFTP account notification text", "wasdiAdminMail": "admin@wasdi.net" } } Notifications Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------+-------------+-------------------------------+ | Property | Type | Description | +==============================+=============+===============================+ | `` | String | Address of the Mercurius | | mercuriusAPIAddress`` | | service (CIMA service API to | | | | send e-mails) | +------------------------------+-------------+-------------------------------+ | ``pwRecoveryMailTitle`` | String | Title of the password | | | | recovery email | +------------------------------+-------------+-------------------------------+ | ``pwRecoveryMailSender`` | String | Sender address for password | | | | recovery emails | +------------------------------+-------------+-------------------------------+ | ``pwRecoveryMailText`` | String | Text content of the password | | | | recovery email | +------------------------------+-------------+-------------------------------+ | ``sftpMailTitle`` | String | Title of the SFTP account | | | | notification email | +------------------------------+-------------+-------------------------------+ | ``sftpManagementMailSender`` | String | Sender address for SFTP | | | | account emails | +------------------------------+-------------+-------------------------------+ | ``sftpMailText`` | String | Text content of the SFTP | | | | account notification email | +------------------------------+-------------+-------------------------------+ | ``wasdiAdminMail`` | String | Declared WASDI administrator | | | | email address | +------------------------------+-------------+-------------------------------+ Data Provider Configuration --------------------------- Each Data Provider is a component able to query and fecth data from a generic external service that can offer one or more data collection. The main business entities involved in this operation are: - Platform: this is the type of data. Usually identified as a Satellite Mission. Platorms are for example Sentinel1, Sentinel2, ENVISat etc. Each Platform, in general, can be found in one or more data providers. - Query Executors / catalogue: the Query executor is the WASDI hierarchy used to query the Data Provider Catalogue - Provider Adapters: Objects used by the launcher to download/import files from an external service. Each Data Provider can support one or more platforms (and each platform can be supported by one or more Data Providers.) All Data Providers have: .name .description .link .user .password Different data providers uses also different parameters. This config has been re-written after that many data providers were already avaiable and is done to support also legacy objects. Is due to control for Each Data Provider witch params are really needed. At the moment of fetching some data, WASDI will try to use the “best” Data Provider: this is an algorithm based on the availability of the data providers and on the Cloud Environment declared by the Node and the data provider itself. For example, a node in Creodias, will try to select the Creodias Data Provider since it assumes it will be faster beeing in the same cloud. A node in LSA Data Center, will probably choose LSA Data Center. In any case, in case of problems, WASDI will try to back-up using all the Data Providers available for that specific Platform. Each Data Provider can also have two additional config files: these are not mandatory. Are meant to be generic JSON files, that may be needed by any specific Data Provider for its own purpose. The two files are: - parserConfig: config file dedicated to the Query Executor - adapterConfig: config file dedicated to the Data Adapter .. code:: json { "dataProviders": [ { "name": "COPERNICUS", "queryExecutorClasspath": "wasdi.dataproviders.CopernicusQueryExecutor", "providerAdapterClasspath": "wasdi.dataproviders.CopernicusProviderAdapter", "pythonScript": "", "description": "Copernicus Data Provider", "link": "https://scihub.copernicus.eu/", "searchListPageSize": "25", "defaultProtocol": "https://", "user": "username", "password": "password", "apiKey": "", "localFilesBasePath": "", "urlDomain": "https://scihub.copernicus.eu/", "connectionTimeout": "10000", "readTimeout": "30000", "adapterConfig": "", "parserConfig": "", "cloudProvider": "COPERNICUS", "supportedPlatforms": ["S1", "S2", "S3"] } ] } Data Provider Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------+-------------+-------------------------------+ | Property | Type | Description | +==============================+=============+===============================+ | ``name`` | String | Name/Code of the data | | | | provider (required to get | | | | QueryExecutors and | | | | ProviderAdapters) | +------------------------------+-------------+-------------------------------+ | ``queryExecutorClasspath`` | String | Full class name and path of | | | | the QueryExecutor | | | | implementation (used to | | | | create the Java class with | | | | reflection) | +------------------------------+-------------+-------------------------------+ | ``providerAdapterClasspath`` | String | Full class name and path of | | | | the ProviderAdapter | | | | implementation (used to | | | | create the Java class with | | | | reflection) | +------------------------------+-------------+-------------------------------+ | ``pythonScript`` | String | Python script path for | | | | External Python Provider | +------------------------------+-------------+-------------------------------+ | ``description`` | String | Description of the data | | | | provider | +------------------------------+-------------+-------------------------------+ | ``link`` | String | Link to the data provider’s | | | | website | +------------------------------+-------------+-------------------------------+ | ``searchListPageSize`` | String | Size for paginated requests | | | | when making WASDI searchList | | | | operations | +------------------------------+-------------+-------------------------------+ | ``defaultProtocol`` | String | Default protocol for data | | | | fetch (“https://” or | | | | “file://”) | +------------------------------+-------------+-------------------------------+ | ``parserConfig`` | String | Path to parser config JSON | | | | file for query conversion | +------------------------------+-------------+-------------------------------+ | ``user`` | String | Username for the data | | | | provider | +------------------------------+-------------+-------------------------------+ | ``password`` | String | Password for the data | | | | provider | +------------------------------+-------------+-------------------------------+ | ``apiKey`` | String | API key for the data provider | +------------------------------+-------------+-------------------------------+ | ` | String | Local base folder for direct | | `localFilesBasePath`` | | file access (when | | | | defaultProtocol is “file://”) | +------------------------------+-------------+-------------------------------+ | ``urlDomain`` | String | API address of the data | | | | provider | +------------------------------+-------------+-------------------------------+ | ``connectionTimeout`` | String | Specific connection timeout | | | | for this data provider | +------------------------------+-------------+-------------------------------+ | ``readTimeout`` | String | Specific read timeout for | | | | this data provider | +------------------------------+-------------+-------------------------------+ | ``adapterConfig`` | String | Path to file with specific | | | | ProviderAdapter | | | | configurations | +------------------------------+-------------+-------------------------------+ | ``cloudProvider`` | String | Code of the cloud provider | | | | where this data provider is | | | | hosted | +------------------------------+-------------+-------------------------------+ | ``supportedPlatforms`` | Array | List of platforms supported | | | [String] | by this data provider | +------------------------------+-------------+-------------------------------+ Catalogue Configuration ----------------------- Catalogues are strongly connected with Data Providers. A Catalogue is a service we can use to query a data collection. As each data provider can support more platforms (data collections), each platform can in general be queried in different catalogues. In this catalogues section, we are intructing WASDI, for each platform, what are the avaiable catalogues. WASDI will try to query the first one first: if there is any problem, WASDI can switch on the other ones in the list. Each name of a catalogue, must match a Data Provider. There is a kind of double link: in the Data Provider, we list all the platforms supported. In this catalogues section, we list for each platform the list of Catalogues (Data Providers) that we can query. The two list in general should match, but is possible for the administrator, for example, to drop from the catalogues list a Data Provider that we do not want to query but, eventually, only to fecth data. .. code:: json { "catalogues": [ { "platform": "S1", "catalogues": ["COPERNICUS", "ONDA", "CREODIAS"] }, { "platform": "S2", "catalogues": ["COPERNICUS", "ONDA", "CREODIAS"] } ] } Catalogue Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------+-------------+-------------------------------+ | Property | Type | Description | +=========================+=============+===============================+ | ``platform`` | String | Code of the Platform Type as | | | | defined by the Platform class | | | | (e.g., “S1”, “S2”, “S3”) | +-------------------------+-------------+-------------------------------+ | ``catalogues`` | Array | List of catalogues supporting | | | [String] | the Platform Type. Each | | | | string is the code of a | | | | QueryExecutor/ProviderAdapter.| | | | The first element has the | | | | highest priority. | +-------------------------+-------------+-------------------------------+ Additional Service Configurations --------------------------------- GeoServer Configuration ^^^^^^^^^^^^^^^^^^^^^^^ .. code:: json { "geoserver": { "address": "http://localhost:8080/geoserver/", "user": "admin", "password": "geoserver", "workspace": "wasdi" } } GeoServer Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +----------------------------------+-------------+----------------------------------+ | Property | Type | Description | +==================================+=============+==================================+ | ``address`` | String | GeoServer server address | | | | (e.g., | | | | “http: | | | | //localhost:8080/geoserver/”) | +----------------------------------+-------------+----------------------------------+ | ``user`` | String | GeoServer username for | | | | authentication | +----------------------------------+-------------+----------------------------------+ | ``password`` | String | GeoServer password for the | | | | above user | +----------------------------------+-------------+----------------------------------+ | ``maxGeotiffDimensionPyramid`` | String | Maximum dimension in MB to | | | | publish single images. Over | | | | this limit, WASDI will create | | | | a pyramid of the image | +----------------------------------+-------------+----------------------------------+ | ``gdalRetileCommand`` | String | GDAL retile command for | | | | pyramid creation (default: | | | | “gdal_retile.py -r bilinear | | | | -levels 4 -ps 2048 2048 -co | | | | TILED=YES”) | +----------------------------------+-------------+----------------------------------+ | ``localDebugPublishBand`` | Boolean | Special debug mode for | | | | PublishBand operation. Forces | | | | input DownloadedFile to be | | | | gathered from database using | | | | /data/wasdi/ instead of real | | | | local folder (default: false) | +----------------------------------+-------------+----------------------------------+ | ``defaultLayerToGetStyleImages`` | String | Default layer used to get | | | | style images (default: | | | | “wasdi:ESA_CCI_LAND_COVER_2015”) | +----------------------------------+-------------+----------------------------------+ SFTP Configuration Structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SFTP Configuration ~~~~~~~~~~~~~~~~~~ .. code:: json { "sftp": { "address": "sftp.wasdi.net", "user": "sftpuser", "password": "sftppassword", "port": 22, "uploadRootPath": "/upload/" } } SFTP Configuration Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------+-------------+-------------------------------+ | Property | Type | Description | +==============================+=============+===============================+ | ``sftpManagementMailSender`` | String | Sender address for SFTP | | | | account emails | +------------------------------+-------------+-------------------------------+ *Note: The JSON example above shows additional properties (``address``, ``user``, ``password``, ``port``, ``uploadRootPath``) that may be used in specific configurations but are not currently defined in the SftpConfig.java class.* S3 Bucket Configuration ~~~~~~~~~~~~~~~~~~~~~~~ .. code:: json { "s3Bucket": { "accessKey": "access-key", "secretAccessKey": "secret-key", "region": "eu-west-1", "bucketName": "wasdi-bucket" } } S3BucketConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^ This configuration is dedicated to the specific access to Ecostress Data, may be not of interest in a personal installation. ============== ====== ============================================= Property Type Description ============== ====== ============================================= ``accessKey`` String Access Key for S3 bucket authentication ``secretKey`` String Secret Key for S3 bucket authentication ``endpoint`` String S3 service endpoint URL ``bucketName`` String Name of the S3 bucket ``folders`` String Folder structure or path within the S3 bucket ============== ====== ============================================= Stripe Configuration ~~~~~~~~~~~~~~~~~~~~ .. code:: json { "stripe": { "secretKey": "sk_test_...", "publicKey": "pk_test_...", "webhookSecret": "whsec_..." } } StripeConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^ +-----------------------+---------------------------+-------------------------------+ | Property | Type | Description | +=======================+===========================+===============================+ | ``apiKey`` | String | Stripe API key for | | | | authentication and payment | | | | processing | +-----------------------+---------------------------+-------------------------------+ | ``products`` | List | List of product-related | | | | configuration entries | +-----------------------+---------------------------+-------------------------------+ StripeProductConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ======== ====== ========================================= Property Type Description ======== ====== ========================================= ``id`` String Product ID in Stripe ``url`` String Product URL for accessing product details ======== ====== ========================================= Storage Usage Control --------------------- .. code:: json { "storageUsageControl": { "enabled": true, "maxDaysBeforeWarning": 7, "maxDaysBeforeDeletion": 30, "adminEmail": "admin@wasdi.net" } } StorageUsageControl Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------------+-------------+-------------------------------+ | Property | Type | Description | +=====================================+=============+===============================+ | ``storageSizeFreeSubscription`` | Long | Maximum storage space (in | | | | bytes) for FREE subscription | | | | plans (default: 20GB) | +-------------------------------------+-------------+-------------------------------+ | ``storageSizeStandardSubscription`` | Long | Maximum storage space (in | | | | bytes) for STANDARD | | | | subscription plans (default: | | | | 100GB) | +-------------------------------------+-------------+-------------------------------+ | ``deletionDelayFromWarning`` | int | Number of days to wait before | | | | proceeding to workspace | | | | deletion after warning email | | | | is sent (default: 10) | +-------------------------------------+-------------+-------------------------------+ | ``isDeletionInTestMode`` | boolean | If true, workspaces with | | | | invalid subscriptions will | | | | not be actually deleted, but | | | | admin notification emails | | | | will be sent instead | +-------------------------------------+-------------+-------------------------------+ | ``warningEmailConfig`` | Warning | Configuration for warning | | | EmailConfig | emails sent to users before | | | | workspace deletion | +-------------------------------------+-------------+-------------------------------+ WarningEmailConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-----------------------+-------------+-------------------------------+ | Property | Type | Description | +=======================+=============+===============================+ | ``title`` | String | Subject line of the warning | | | | email (default: “Storage | | | | exceeded in WASDI”) | +-----------------------+-------------+-------------------------------+ | ``message`` | String | Body content of the warning | | | | email with placeholders for | | | | user info, storage size, | | | | limit, and warning delay | +-----------------------+-------------+-------------------------------+ Skin Configuration ------------------ .. code:: json { "skins": [ { "name": "wasdi", "description": "Default WASDI Skin", "logoPath": "/assets/logos/wasdi.png", "primaryColor": "#1976d2" } ] } SkinConfig Properties ^^^^^^^^^^^^^^^^^^^^^ +-----------------------+-------------+-------------------------------+ | Property | Type | Description | +=======================+=============+===============================+ | ``name`` | String | Name identifier for the skin | | | | (default: “wasdi”) | +-----------------------+-------------+-------------------------------+ | ``logoImage`` | String | Path to the logo image file | | | | (default: | | | | “ | | | | /assets/icons/logo-only.svg”) | +-----------------------+-------------+-------------------------------+ | ``logoText`` | String | Path to the logo text/name | | | | image file (default: | | | | “ | | | | /assets/icons/logo-name.svg”) | +-----------------------+-------------+-------------------------------+ | ``helpLink`` | String | URL link to help | | | | documentation (default: | | | | “https://wasd | | | | i.readthedocs.io/en/latest/”) | +-----------------------+-------------+-------------------------------+ | ``supportLink`` | String | URL link to support channels | | | | (default: | | | | “ht | | | | tps://discord.gg/FkRu2GypSg”) | +-----------------------+-------------+-------------------------------+ | ``brandMainColor`` | String | Primary brand color in hex | | | | format (default: “#43526B”) | +-----------------------+-------------+-------------------------------+ |``brandSecondaryColor``| String | Secondary brand color in hex | | | | format (default: “#009036”) | +-----------------------+-------------+-------------------------------+ | ``defaultCategories`` | ArrayList | List of default categories | | | | for the skin | +-----------------------+-------------+-------------------------------+ Node Score Configuration ------------------------ .. code:: json { "nodeScore": { "minTotalMemoryGBytes": 30 } } NodeScoreConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^ +--------------------------+-------------+-------------------------------+ | Property | Type | Description | +==========================+=============+===============================+ | ``minTotalMemoryGBytes`` | int | Minimum total memory in GB | | | | required for node scoring | | | | (default: 30) | +--------------------------+-------------+-------------------------------+ SNAP Configuration ------------------ .. code:: json { "snap": { "auxPropertiesFile": "/usr/lib/wasdi/launcher/snap.auxdata.properties", "launcherLogActive": true, "webLogActive": true, "launcherLogFile": "/var/log/wasdi/snap-launcher.log", "webLogFile": "/var/log/wasdi/snap-web.log", "launcherLogLevel": "SEVERE", "webLogLevel": "SEVERE" } } SnapConfig Properties ^^^^^^^^^^^^^^^^^^^^^ +-----------------------+-------------+-------------------------------+ | Property | Type | Description | +=======================+=============+===============================+ | ``auxPropertiesFile`` | String | Full path to SNAP auxiliary | | | | properties file (default: | | | | “/usr/lib/wasdi/launc | | | | her/snap.auxdata.properties”) | +-----------------------+-------------+-------------------------------+ | ``launcherLogActive`` | boolean | Flag to activate SNAP logs in | | | | the launcher (default: true) | +-----------------------+-------------+-------------------------------+ | ``webLogActive`` | boolean | Flag to activate SNAP logs in | | | | the web server (default: | | | | true) | +-----------------------+-------------+-------------------------------+ | ``launcherLogFile`` | String | Full path to SNAP launcher | | | | log file | +-----------------------+-------------+-------------------------------+ | ``webLogFile`` | String | Full path to SNAP web server | | | | log file | +-----------------------+-------------+-------------------------------+ | ``launcherLogLevel`` | String | SNAP launcher log level | | | | (default: “SEVERE”) | +-----------------------+-------------+-------------------------------+ | ``webLogLevel`` | String | SNAP web server log level | | | | (default: “SEVERE”) | +-----------------------+-------------+-------------------------------+ Scheduler Configuration ----------------------- This section configures the WASDI Scheduler. For each operation type, it can add a queue. Is up to the admin to refine the configuration given the capabilities of the node where it is installed. All the operations that are not explicitly assigned, will go in the default queue. The admin can then add all the needed queues. .. code:: json { "scheduler": { "processingThreadWaitStartMS": "2000", "processingThreadSleepingTimeMS": "2000", "launcherPath": "/opt/wasdi/launchers/launcher.jar", "javaExe": "java", "killCommand": "kill -9", "maxQueue": "50", "timeoutMs": "300000", "lastStateChangeDateOrderBy": -1, "sometimesCheckCounter": 30, "watchDogCounter": 30, "activateWatchDog": true, "redirectLauncherOutputs": false, "launcherOutputPath": "", "useVirtualNodeCode": false, "defaultFairRoundRobin": false, "defaultFairRoundRobinMaxProcessesCount": 3, "defaultFairRoundRobinParentId": false, "defaultFairRoundRobinParentIdMaxProcessesCount": 3, "defaultSpecialWaitCondition": false, "defaultMaxWaitingQueue": 100, "emergencyFallbackEveryNCycles": 5, "schedulers": [ { "name": "default", "maxQueue": "50", "timeoutMs": "300000", "opTypes": "RUNPROCESSOR,RUNIDL,RUNMATLAB", "opSubType": "", "enabled": "true", "fairRoundRobin": false, "fairRoundRobinMaxProcessesCount": 3, "fairRoundRobinParentId": false, "fairRoundRobinParentIdMaxProcessesCount": 3, "specialWaitCondition": false, "maxWaitingQueue": 100 }, { "name": "download", "maxQueue": "10", "timeoutMs": "600000", "opTypes": "DOWNLOAD", "opSubType": "", "enabled": "true", "specialWaitCondition": true, "maxWaitingQueue": 50 } ] } } SchedulerConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^ These are the configurations of the default scheduler +------------------------------------------------------+-----------+----------------------------------------------+ | Property | Type | Description | +======================================================+===========+==============================================+ | ``processingThreadWaitStartMS`` | String | Number of milliseconds to wait after a | | | | process is started | +------------------------------------------------------+-----------+----------------------------------------------+ | ``processingThreadSleepingTimeMS`` | String | Number of milliseconds to sleep between | | | | scheduler cycles | +------------------------------------------------------+-----------+----------------------------------------------+ | ``launcherPath`` | String | Full path of Launcher jar file | +------------------------------------------------------+-----------+----------------------------------------------+ | ``javaExe`` | String | Local Java command line executable | +------------------------------------------------------+-----------+----------------------------------------------+ | ``killCommand`` | String | OS kill command for terminating processes | +------------------------------------------------------+-----------+----------------------------------------------+ | ``maxQueue`` | String | Default maximum queue size | +------------------------------------------------------+-----------+----------------------------------------------+ | ``timeoutMs`` | String | Default queue timeout in milliseconds | +------------------------------------------------------+-----------+----------------------------------------------+ | ``lastStateChangeDateOrderBy`` | int | Direction to order process workspaces in | | | | scheduler queue (default: -1) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``sometimesCheckCounter`` | int | Number of cycles before starting periodic | | | | checks (default: 30) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``watchDogCounter`` | int | Counter for deadlock detection when only | | | | waiting processes exist (default: 30) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``activateWatchDog`` | boolean | Flag to activate or deactivate the watch | | | | dog functionality (default: true) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``redirectLauncherOutputs`` | boolean | If true, asks launcher to redirect its | | | | output to file | +------------------------------------------------------+-----------+----------------------------------------------+ | ``launcherOutputPath`` | String | Destination path used when launcher output | | | | redirection is enabled (default: "") | +------------------------------------------------------+-----------+----------------------------------------------+ | ``useVirtualNodeCode`` | boolean | Enables virtual node code filtering for | | | | scheduler execution (default: false) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``defaultFairRoundRobin`` | boolean | Enables fair round robin per user in | | | | CREATED list (default: false) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``defaultFairRoundRobinMaxProcessesCount`` | int | Max consecutive processes per user when | | | | fair round robin is active (default: 3) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``defaultFairRoundRobinParentId`` | boolean | Enables second round robin by parentId | | | | inside selected user queue (default: false) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``defaultFairRoundRobinParentIdMaxProcessesCount`` | int | Max consecutive processes per parentId | | | | group when parentId round robin is active | | | | (default: 3) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``defaultSpecialWaitCondition`` | boolean | Enables special wait condition to consider | | | | waiting queue before triggering processes | | | | (default: false) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``defaultMaxWaitingQueue`` | int | Max waiting processes admitted before | | | | breaking FIFO rules (default: 100) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``emergencyFallbackEveryNCycles`` | int | In emergency mode, fallback to default | | | | CREATED candidate every N cycles when no | | | | unblock candidate is found (default: 5) | +------------------------------------------------------+-----------+----------------------------------------------+ | ``schedulers`` | ArrayList | List of configured scheduler queues | +------------------------------------------------------+-----------+----------------------------------------------+ SchedulerQueueConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Each of these, is a specific queue. +----------------------------------------------+---------+----------------------------------------------+ | Property | Type | Description | +==============================================+=========+==============================================+ | ``name`` | String | Name identifier of the queue | +----------------------------------------------+---------+----------------------------------------------+ | ``maxQueue`` | String | Maximum number of elements allowed in this | | | | queue | +----------------------------------------------+---------+----------------------------------------------+ | ``timeoutMs`` | String | Queue timeout in milliseconds for this | | | | specific queue | +----------------------------------------------+---------+----------------------------------------------+ | ``opTypes`` | String | Comma-separated operation types supported | | | | by this queue | +----------------------------------------------+---------+----------------------------------------------+ | ``opSubType`` | String | Operation subtype (requires opTypes to | | | | contain only one operation) | +----------------------------------------------+---------+----------------------------------------------+ | ``enabled`` | String | Flag to enable or disable this queue | +----------------------------------------------+---------+----------------------------------------------+ | ``specialWaitCondition`` | boolean | Flag to apply special wait condition | | | | considering waiting queue (default: false) | +----------------------------------------------+---------+----------------------------------------------+ | ``maxWaitingQueue`` | int | Maximum waiting processes before breaking | | | | FIFO rules (default: 100) | +----------------------------------------------+---------+----------------------------------------------+ | ``fairRoundRobin`` | boolean | Enables fair round robin per user in | | | | CREATED list (default: false) | +----------------------------------------------+---------+----------------------------------------------+ | ``fairRoundRobinMaxProcessesCount`` | int | Max consecutive processes per user when | | | | fair round robin is active (default: 3) | +----------------------------------------------+---------+----------------------------------------------+ | ``fairRoundRobinParentId`` | boolean | Enables second round robin by parentId | | | | inside selected user queue (default: false) | +----------------------------------------------+---------+----------------------------------------------+ | ``fairRoundRobinParentIdMaxProcessesCount`` | int | Max consecutive processes per parentId | | | | group when parentId round robin is active | | | | (default: 3) | +----------------------------------------------+---------+----------------------------------------------+ Scheduler Defaults and Queue Overrides ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The scheduler-level properties prefixed with ``default`` are global defaults applied to queue behavior. If a queue in ``schedulers`` defines the corresponding queue-level property, that queue value takes precedence for that queue. Otherwise, the scheduler default is used. Main mappings: - ``defaultFairRoundRobin`` -> ``fairRoundRobin`` - ``defaultFairRoundRobinMaxProcessesCount`` -> ``fairRoundRobinMaxProcessesCount`` - ``defaultFairRoundRobinParentId`` -> ``fairRoundRobinParentId`` - ``defaultFairRoundRobinParentIdMaxProcessesCount`` -> ``fairRoundRobinParentIdMaxProcessesCount`` - ``defaultSpecialWaitCondition`` -> ``specialWaitCondition`` - ``defaultMaxWaitingQueue`` -> ``maxWaitingQueue`` Load Balancer Configuration --------------------------- .. code:: json { "loadBalancer": { "includeMainClusterAsNode": false, "diskOccupiedSpaceMaxPercentage": 90, "metricsMaxAgeSeconds": 600, "minTotalMemoryGBytes": 30, "activateMetrics": true } } LoadBalancerConfig Properties ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------------+-------------+-------------------------------+ | Property | Type | Description | +====================================+=============+===============================+ | ``includeMainClusterAsNode`` | boolean | Flag to include main node in | | | | computational node evaluation | | | | (default: false) | +------------------------------------+-------------+-------------------------------+ | ``diskOccupiedSpaceMaxPercentage`` | int | Maximum percentage of disk | | | | space occupied to consider a | | | | node available (default: 90) | +------------------------------------+-------------+-------------------------------+ | ``metricsMaxAgeSeconds`` | int | Maximum age in seconds for | | | | node metrics before | | | | considering node down | | | | (default: 600) | +------------------------------------+-------------+-------------------------------+ | ``minTotalMemoryGBytes`` | int | Minimum RAM in GB required | | | | for a node to avoid low | | | | performance penalty (default: | | | | 30) | +------------------------------------+-------------+-------------------------------+ | ``activateMetrics`` | boolean | Flag to enable/disable | | | | metrics-based node selection | | | | (default: true) | +------------------------------------+-------------+-------------------------------+ Configuration File Location --------------------------- The configuration file is typically located at: - ``/opt/wasdi/config/wasdiConfig.json`` (production) - ``/data/wasdi/config/wasdiConfig.json`` (development) Loading the Configuration ------------------------- The configuration is loaded automatically when WASDI components start up using the ``WasdiConfig.readConfig(String configFilePath)`` method. The loaded configuration is available statically through ``WasdiConfig.Current``. Environment-Specific Configurations ----------------------------------- Different environments (development, staging, production) should have separate configuration files with appropriate values for: - Database connections - External service URLs - Authentication settings - File system paths - Docker registry settings - API keys and secrets Security Considerations ----------------------- - Store sensitive information (passwords, API keys) securely - Use environment variables or secure vaults for production secrets - Regularly rotate authentication credentials - Restrict file system permissions on the configuration file - Use HTTPS for all external service communications Validation ---------- The configuration is validated when loaded. Missing required properties or invalid values will cause WASDI components to fail during startup with appropriate error messages.