GitLab

* Add tools to connect to HSQLDB CLI

* Apply reviews

* Add .bat file and adapt the java path

* Small typo: add -jar

install.sh : Remove empty line before shebang

Fix selection log max size

This setting can overwrite pa.scheduler.job.logs.max.size=10MB from config/scheduler/settings.ini
Accordingly, they must be both equal

add expert-ds and citizen-ds users

Fix bad mistake in housekeeping

Conversion from long to integer implied to set SCHEDULER_REMOVED_JOB_DELAY = 0 when the configured value is bigger than 24 days (due to MAX_INTEGER reached).

Use now the appropriate getValueAsLong() method.

Generic Info REMOVE_DELAY : change format

 - use Tools.parsePeriod instead of Tools.formatDate for a more comprehensive duration format

Generic Info REMOVE_DELAY to remove a job after a configurable time

- Add functional test for housekeeping

- Also, fix issue with ServerJobAndTaskLogs while removing precious logs (access to SchedulerSpacesSupport from a non-active object context).
Fix minor issues appearing in the code

Optimize JobData and TaskData queries

 - discontinue support of pa.scheduler.job.removeFromDataBase=false, which implied executing joined tables queries constantly (decreases performance and disallow some indexing)
 - add 3 indexes in JobData (submit_time, scheduled_time_for_removal, and job_id_status : composite index used to optimize scheduler portal default list jobs query)
 - getTaskCounts : avoid joined table query by considering only task statuses.
 - getFinishedTaskCount : fix incomplete list of finished task statuses
 - getPendingTaskCount : add in-error status (similar to pause)
 - adapt db tests
 - add owner column in TaskData to avoid join
 - initialize TaskData.owner using JobData.owner when recovering an existing database
 - TaskData : replace task.jobData.id by task.id.jobId to avoid implicit cross join
 - TaskDBUtils : supress removeTime usage and replace jobData.owner by taskData.owner
 - BaseServiceTest : add SQL debug info
 - MockSchedulingInfrastructure : print unexpected exceptions

Handle the case where the value of a resultMap entry is null

ZipUtils: do not perform compression in memory

Current implementation was performing compression using a ByteArrayOutputStream in memory.
Problem: When the amount of data to compress is big, this clearly exhaust the memory usage.

Solution : perform compression directly inside the zip file destination.

synchronize fork

MemoryLeak: do not keep finished jobs state in memory

 - SchedulerStateImpl : use a maximum size for finished jobs. As test is using the getState method a lot, it is necessary to keep some retro-compatibility. The default number of jobs kept in SchedulerStateImpl is 1000, but can be configured through a JVM property.
 - SchedulerFrontendState : delete job from jobsMap when the job is finished.

Make FileAppender pattern configurable

Add a new rm property which allows to define a conversion pattern for all FileAppenders (job/task server logs)

Add poolName for hikari pools

Fix NodeRecovery eligible status

 - use of setEligibleNodesToRecover in NodesRecoveryManager means that if several node sources are present, the set of eligible nodes is overwritten.
 - replaced by a list add method
 - additionnally, in proactive-scheduler service definition fix the node name parameter.

Remove and comment recursive algorithm in InternalJob

Replaced by a simple iterative search

After analysis of the code, it tries to find an InternalTask based on its name, but to do so, it does a complex  potentially exponential recursive algorithm. This code hanged a task termination on a workflow containing a loop which had around 1400 tasks.

For now I comment the previous code to keep its track.

Stop the scheduler only in case of jython deadlocks

It has been observed recently that shutdown of the scheduler server occurs in case of server lag, not due to python deadlocks. The goal of this change is to trigger a shutdown only when the cause is due to jython and not something else.

synchronize fork

Remove empty lines from job logs full