GitLab

Fix bad mistake in housekeeping

Conversion from long to integer implied to set SCHEDULER_REMOVED_JOB_DELAY = 0 when the configured value is bigger than 24 days (due to MAX_INTEGER reached).

Use now the appropriate getValueAsLong() method.

Generic Info REMOVE_DELAY : change format

 - use Tools.parsePeriod instead of Tools.formatDate for a more comprehensive duration format

Generic Info REMOVE_DELAY to remove a job after a configurable time

- Add functional test for housekeeping

- Also, fix issue with ServerJobAndTaskLogs while removing precious logs (access to SchedulerSpacesSupport from a non-active object context).
Fix minor issues appearing in the code

Optimize JobData and TaskData queries

 - discontinue support of pa.scheduler.job.removeFromDataBase=false, which implied executing joined tables queries constantly (decreases performance and disallow some indexing)
 - add 3 indexes in JobData (submit_time, scheduled_time_for_removal, and job_id_status : composite index used to optimize scheduler portal default list jobs query)
 - getTaskCounts : avoid joined table query by considering only task statuses.
 - getFinishedTaskCount : fix incomplete list of finished task statuses
 - getPendingTaskCount : add in-error status (similar to pause)
 - adapt db tests
 - add owner column in TaskData to avoid join
 - initialize TaskData.owner using JobData.owner when recovering an existing database
 - TaskData : replace task.jobData.id by task.id.jobId to avoid implicit cross join
 - TaskDBUtils : supress removeTime usage and replace jobData.owner by taskData.owner
 - BaseServiceTest : add SQL debug info
 - MockSchedulingInfrastructure : print unexpected exceptions

Handle the case where the value of a resultMap entry is null

ZipUtils: do not perform compression in memory

Current implementation was performing compression using a ByteArrayOutputStream in memory.
Problem: When the amount of data to compress is big, this clearly exhaust the memory usage.

Solution : perform compression directly inside the zip file destination.

synchronize fork

MemoryLeak: do not keep finished jobs state in memory

 - SchedulerStateImpl : use a maximum size for finished jobs. As test is using the getState method a lot, it is necessary to keep some retro-compatibility. The default number of jobs kept in SchedulerStateImpl is 1000, but can be configured through a JVM property.
 - SchedulerFrontendState : delete job from jobsMap when the job is finished.

Make FileAppender pattern configurable

Add a new rm property which allows to define a conversion pattern for all FileAppenders (job/task server logs)

Add poolName for hikari pools

Fix NodeRecovery eligible status

 - use of setEligibleNodesToRecover in NodesRecoveryManager means that if several node sources are present, the set of eligible nodes is overwritten.
 - replaced by a list add method
 - additionnally, in proactive-scheduler service definition fix the node name parameter.

Remove and comment recursive algorithm in InternalJob

Replaced by a simple iterative search

After analysis of the code, it tries to find an InternalTask based on its name, but to do so, it does a complex  potentially exponential recursive algorithm. This code hanged a task termination on a workflow containing a loop which had around 1400 tasks.

For now I comment the previous code to keep its track.

Stop the scheduler only in case of jython deadlocks

It has been observed recently that shutdown of the scheduler server occurs in case of server lag, not due to python deadlocks. The goal of this change is to trigger a shutdown only when the cause is due to jython and not something else.

synchronize fork

Remove empty lines from job logs full

Empty lines was created by log4j messages already containing end of lines characters.
This fix makes sure that each log message terminates with a single end-of-line (in Log4JTaskLogs, thanks to StringUtils.stripEnd).
It also makes sure that no blank lines are added between individual task logs (changes in SchedulerStateRest).

Change content-type produced by log outputs

 - log outputs are in general not valid json as it simply returns the contents of the log as plain text
 - json is returned only in case of errors (wrong job id for example)
 - plain text as application/json cannot be forcefully parsed by angular/js. So in order to use these requests from angular, it is necessary to advertise them as text/plain
 - did not modify the live logs output (for now)

Add api links to ProActive frontpage

synchronize fork

* update welcome page

* remove assets

* add all resources

* remove useless files

* remove additional slash

Fix dataspace warnings in task logs

Warnings or errors when transferring files using dataspaces used to appear in task logs but these logs disappeared (regression).
This commit fixes the issue.
Addressed reviews