Improving data sources

Created by: ikedas

NOTE: Please discuss/report bugs on #693 (closed).

---

Changes in code:

• Introducing new classes Sympa::Request::Handler::include, Sympa::DataSource and its subclasses. Sympa::Datasource was deprecated.

Changes in behavior:

• Now data sources won't overwrite display name (gecos) of existing subscriber: It is assigned only when a new user is added (included) from data source, or if it is added (subscribed) by users. Previously, some (all, with very earlier version) data sources overwrote.
• Outdated users who have been included from data source will be expired only when all data sources of the list succeeds inclusion. Previously, users were delayed expiration when corresponding data source failed synchronization: In order to do that, entire subscribers had to be loaded in memory (this bug has been pointed out by #275). By this change, if there are data sources with nosync_time_ranges set, deletion of outdated users can be delayed 24 hours at most.
• Names of data sources will no longer be shown in subscriber table of web interface.
• All names of data sources will no longer be shown in subscriber table of web interface. Only one that was actually included will be shown.

Other changes:

• Database schema: included_* and include_sources_* in subscriber_table and admin_table were deprecated. inclusion_* and inclusion_ext_* fields will be used instead. inclusion_* holds the last time of confirmation of inclusion. Additionally, inclusion_ext_* holds the time from "external" data sources. "External" means that it is not include_sympa_list or not including list on local domain (this data source should be treated specially to detect inclusion loop). Table content will be updated during upgrading process.
• include_voot_group has not been implemented, and it was invalidated.

Improvements and bug fixes:

• Memory usage:
  • include_sql_query and include_sql_ca stores inclusion data in temporary file so that big data sources will be handled safely.
  • See also "Changes in behavior" above.
• include_ldap_2level_ca data source was implemented. However it needs more tests.
• include_remote_sympa_list data source:
  • HTTP/1.1 is supported thanks to LWP::UserAgent.
  • Now it has user, passwd, timeout, ssl_version, ssl_ciphers and ca_verify parameters.
  • host, port and path parameters were obsoleted. Use url instead.
  • cert parameter was obsoleted.
• include_remote_file data source:
  • Now it has timeout, ssl_version, ssl_ciphers, ca_verify parameters.
  • It may use client certificate and key in cert.pem and private_key files (as include_remote_sympa_list does).
• include_ldap_query and include_ldap_ca data sources did not implement "regex" option of "selection" subparameter. Now they support it.
• Now include_remote_file, include_sympa_list and include_remote_sympa_list paragraphs also may have nosync_time_ranges parameter. No reason they don't have it. include_file still doesn't have it.
• Consistencies on parameter names (older parameter names are also available):
  • include_remote_sympa_list: New parameter url replaced host, port and path.
  • include_ldap_query, include_ldap_2level_query, include_ldap_ca & include_ldap_2level_ca: bind_dn & bind_password replaced user & passwd, respectively.
  • include_sql_query & include_sql_ca: db_host, db_options, db_user & db_passwd replaced host, connect_options, user & passwd, respectively.
• [bug] include_remote_file should limit protocols to HTTP, HTTPS and FTP.
• [bug] LDAP multiple values in custom attribute with include_ldap_ca was broken.

Known bug:

• If data source is a incude_sympa_list data source with the list included from the other external data source(s), it will be treated as non-external and move_user request on corresponding users will be allowed.