Features and Improvements in ArangoDB 2.5

    The built-in version of V8 has been upgraded from 3.29.54 to 3.31.74.1.This allows activating additional ES6 (also dubbed Harmony or ES.next) features in ArangoDB, both in the ArangoShell and the ArangoDB server. They can beused for scripting and in server-side actions such as Foxx routes, traversalsetc.

    The following additional ES6 features become available in ArangoDB 2.5 by default:

    • iterators and generators
    • template strings
    • enhanced object literals
    • enhanced numeric literals
    • block scoping with and constant variables using const (note: constant variables require using strict mode, too)
    • additional string methods (such as startsWith, repeat etc.)

    Hash and skiplist indexes can optionally be made sparse. Sparse indexes exclude documentsin which at least one of the index attributes is either not set or has a value of null.

    As such documents are excluded from sparse indexes, they may contain fewer documents thantheir non-sparse counterparts. This enables faster indexing and can lead to reduced memoryusage in case the indexed attribute does occur only in some, but not all documents of the collection. Sparse indexes will also reduce the number of collisions in non-unique hashindexes in case non-existing or optional attributes are indexed.

    In order to create a sparse index, an object with the attribute sparse can be added tothe index creation commands:

    Note that in place of the above specialized index creation commands, it is recommended to usethe more general index creation command ensureIndex:

    1. db.collection.ensureIndex({ type: "hash", sparse: true, unique: true, fields: [ attributeName ] });
    2. db.collection.ensureIndex({ type: "skiplist", sparse: false, unique: false, fields: [ "a", "b" ] });

    When not explicitly set, the sparse attribute defaults to false for new hash or skiplist indexes.

    This causes a change in behavior when creating a unique hash index without specifying the sparse flag: in 2.4, unique hash indexes were implicitly sparse, always excluding null values. There was no option to control this behavior, and sparsity was neither supported for non-uniquehash indexes nor skiplists in 2.4. This implicit sparsity of unique hash indexes was consideredan inconsistency, and therefore the behavior was cleaned up in 2.5. As of 2.5, indexes willonly be created sparse if sparsity is explicitly requested. Existing unique hash indexes from 2.4 or before will automatically be migrated so they are still sparse after the upgrade to 2.5.

    Geo indexes are implicitly sparse, meaning documents without the indexed location attribute orcontaining invalid location coordinate values will be excluded from the index automatically. Thisis also a change when compared to pre-2.5 behavior, when documents with missing or invalidcoordinate values may have caused errors on insertion when the geo index’ unique flag was setand its ignoreNull flag was not. This was confusing and has been rectified in 2.5. The methodensureGeoConstraint() now does the same as ensureGeoIndex(). Furthermore, the attributesconstraint, unique, ignoreNull and sparse flags are now completely ignored when creatinggeo indexes.

    The same is true for fulltext indexes. There is no need to specify non-uniqueness or sparsity for geo or fulltext indexes.

    As sparse indexes may exclude some documents, they cannot be used for every type of query.Sparse hash indexes cannot be used to find documents for which at least one of the indexed attributes has a value of null. For example, the following AQL query cannot use a sparse index, even if one was created on attribute attr:

    If the lookup value is non-constant, a sparse index may or may not be used, depending onthe other types of conditions in the query. If the optimizer can safely determine thatthe lookup value cannot be null, a sparse index may be used. When uncertain, the optimizerwill not make use of a sparse index in a query in order to produce correct results.

    For example, the following queries cannot use a sparse index on attr because the optimizerwill not know beforehand whether the comparison values for doc.attr will include null:

    2.   FILTER doc.attr == SOME_FUNCTION(...) 
    3.   RETURN doc
    5. FOR other IN otherCollection 
    6.     FILTER doc.attr == other.attr 
    7.     RETURN doc

    Selectivity estimates

    Indexes of type primary, edge and hash now provide selectivity estimates. These will be used by the AQL query optimizer when deciding about index usage. Using selectivityestimates can lead to faster query execution when more selective indexes are used.

    The selectivity estimates are also returned by the GET /_api/index REST API methodin a sub-attribute selectivityEstimate for each index that supports it. Thisattribute will be omitted for indexes that do not provide selectivity estimates.If provided, the selectivity estimate will be a numeric value between 0 and 1.

    Selectivity estimates will also be reported in the result of collection.getIndexes()for all indexes that support this. If no selectivity estimate can be determined for an index, the attribute selectivityEstimate will be omitted here, too.

    The web interface also shows selectivity estimates for each index that supports this.

    Currently the following index types can provide selectivity estimates:

    • primary index
    • edge index
    • hash index (unique and non-unique)

    No selectivity estimates will be provided for indexes when running in cluster mode.

    Sort removal

    The AQL optimizer rule “use-index-for-sort” will now remove sorts also in case a non-sortedindex (e.g. a hash index) is used for only equality lookups and all sort attributes are covered by the equality lookup conditions.

    For example, in the following query the extra sort on doc.value will be optimized awayprovided there is an index on doc.value):

    The AQL optimizer rule “use-index-for-sort” now also removes sort in case the sort criteriaexcludes the left-most index attributes, but the left-most index attributes are usedby the index for equality-only lookups.

    For example, in the following query with a skiplist index on value1, value2, the sort canbe optimized away:

    1. FOR doc IN collection 
    2.   FILTER doc.value1 == 1 
    3.   SORT doc.value2 
    4.   RETURN doc

    Constant attribute propagation

    The new AQL optimizer rule propagate-constant-attributes will look for attributes that areequality-compared to a constant value, and will propagate the comparison value into otherequality lookups. This rule will only look inside FILTER conditions, and insert constantvalues found in s, too.

    For example, the rule will insert 42 instead of i.value in the second FILTER of the following query:

    Interleaved processing

    The optimizer will now inspect AQL data-modification queries and detect if the query’s data-modification part can run in lockstep with the data retrieval part of the query, or if the data retrieval part must be executed and completed first before the data-modification can start.

    The filtered attribute was added to AQL query execution statistics. The value of thisattribute indicates how many documents were filtered by FilterNodes in the AQL query.Note that IndexRangeNodes can also filter documents by selecting only the required rangesfrom the index. The filtered value will not include the work done by IndexRangeNodes, but only the work performed by FilterNodes.

    Dynamic attribute names in AQL object literals

    This change allows using arbitrary expressions to construct attribute names in objectliterals specified in AQL queries. To disambiguate expressions and other unquoted attribute names, dynamic attribute names need to be enclosed in brackets ([ and ]).

    Example:

    1. FOR i IN 1..100
    2.   RETURN { [ CONCAT('value-of-', i) ] : i }

    AQL functions

    The following AQL functions were added in 2.5:

    • MD5(value): generates an MD5 hash of value
    • SHA1(value): generates an SHA1 hash of value
    • RANDOM_TOKEN(length): generates a random string value of the specified length

    Thanks to our user feedback we learned that Foxx is a powerful, yet rather complicated concept.With 2.5 we made it less complicated while keeping all its strength.That includes a rewrite of the documentation as well as some code changes as follows:

    Moved Foxx applications to a different folder.

    Until 2.4 Foxx apps were stored in the following folder structure:<app-path>/databases/<dbname>/<appname>:<appversion>.This caused some trouble as apps where cached based on name and version and updates did not apply.Also the path on filesystem and the app’s access URL had no relation to one another.Now the path on filesystem is identical to the URL (except the appended APP):<app-path>/_db/<dbname>/<mointpoint>/APP

    Rewrite of Foxx routing

    The routing of Foxx has been exposed to major internal changes we adjusted because of user feedback.This allows us to set the development mode per mount point without having to change paths and holdapps at separate locations.

    The development mode used until 2.4 is gone. It has been replaced by a much more mature version.This includes the deprecation of the javascript.dev-app-path parameter, which is useless since 2.5.Instead of having two separate app directories for production and development, apps now reside in one place, which is used for production as well as for development.Apps can still be put into development mode, changing their behavior compared to production mode.Development mode apps are still reread from disk at every request, and still they ship more debug output.

    This change has also made the startup options —javascript.frontend-development-mode and —javascript.dev-app-path obsolete. The former option will not have any effect when set, and thelatter option is only read and used during the upgrade to 2.5 and does not have any effects later.

    Foxx install process

    Installing Foxx apps has been a two step process: import them into ArangoDB and mount them at aspecific mount point. These operations have been joined together. You can install an app at onemount point, that’s it. No fetch, mount, unmount, purge cycle anymore. The commands have been simplified to just:

    • install: get your Foxx app up and running
    • uninstall: shut it down and erase it from disk

    Foxx error output

    Until 2.4 the errors produced by Foxx were not optimal. Often, the error message was justunable to parse manifest and contained only an internal stack trace.In 2.5 we made major improvements there, including a much more fine grained error output thathelps you debug your Foxx apps. The error message printed is now much closer to its source and should help you track it down.

    Also we added the default handlers for unhandled errors in Foxx apps:

    • You will get a proper error message when having an uncaught error appears in any app route

    In production mode the messages above will NOT contain any information about your Foxx internalsand are safe to be exposed to third party users.In development mode the messages above will contain the stacktrace (if available), making it easier foryour in-house devs to track down errors in the application.

    Foxx console

    Foxx requests

    We added org/arangodb/request module, which provides a simple API for making HTTP requeststo external services. This is enables Foxx to be directly part of a micro service architecture.