Log Levels

    • FATAL
    • ERROR
    • INFO
    • DEBUG
    • TRACE

    For each log topic one can configure the lowest level which is actually logged.For example, if one sets the log level to for some log topic,one only sees messages of level and above ( and ).

    See an example of how toin the Administration chapter.

    Fatal errors are the most severe errors and only occur if a service or applicationcan not recover safely from an abnormal state, which forces it to shut down.

    Typically, a fatal error only occurs once in the process lifetime,so if the log file is tied to the process, this is typicallythe last message in the log. There might be a few exceptions to thisrule, where it makes more sense to keep the server running, for exampleto be able to diagnose the problem better.

    We reserve this error type for the following events:

    • crucial files/folders are missing or inaccessible during startup
    • overall application or system failure with a serious danger ofdata corruption or loss (the following shutdown is intended to preventpossible or further data loss)

    Recommendation:Fatal errors should be investigated immediately by a system administrator.

    ERROR

    If a problem is encountered which is fatal to some operation, but not forthe service or the application as a whole, then an error is logged.

    • missing data
    • a required file can’t be opened
    • incorrect connection strings

    If some operation is automatically retried and eventually succeeds,no error will be written to the log. Therefore, if an error is logged thenit should be taken seriously as it may require user intervention to solve.

    Note that in any distributed system, temporary failures of network connectionsor certain servers or services can and will happen. Most systems will toleratesuch failures and retry for some time, but will eventually run out of patience,give up and fail the operation one level up.

    Recommendation:A system administrator should be notified automatically to investigate the error.By filtering the log to look at errors (and other logged events above)one can determine the error frequency and quickly identify the initial failurethat might have resulted in a cascade of additional errors.

    A warning is triggered by anything that can potentially causeapplication oddities, but from which the system recovers automatically.

    Examples of events which lead to warnings:

    • switching from a primary to backup server
    • retrying an operation
    • missing secondary data
    • things running inefficiently(in particular slow queries and bad system settings)

    Certain warnings are logged at startup time only, such as startup optionvalues which lie outside the recommended range.

    These might be problems, or might not. For example, expected transientenvironmental conditions such as short loss of network or databaseconnectivity are logged as warnings, not errors. Viewing a log filteredto show only warnings and errors may give quick insight into earlyhints at the root cause of subsequent errors.

    INFO

    Info messages are generally useful information to log to betterunderstand what state the system is in. One will usually want tohave info messages available but does usually not care about themunder normal circumstances.

    Informative messages are logged in events like:

    • successful initialization
    • services starting or stopping
    • successful completion of significant transactions
    • configuration assumptions

    Viewing log entries of severity info and above should give a quick overviewof major state changes in the process providing top-level context forunderstanding any warnings or errors that also occur. Logging info levelmessages and above will usually not spam anything beyond good readability.

    Recommendation:Usually good to have, but one does not have to look at info level messagesunder normal circumstances.

    Information which is helpful to ArangoDB developers as well as to otherpeople like system administrators to diagnose an application or serviceis logged as debug message.

    Debug messages make software much more maintainable but require somediligence because the value of individual debug statements may changeover time as programs evolve. The best way to achieve this is by gettingthe development team in the habit of regularly reviewing logs as a standardpart of troubleshooting reported issues. We encourage our teams toprune out messages that no longer provide useful context and to addmessages where needed to understand the context of subsequent messages.

    Recommendation:Debug level messages are usually switched off, but one can switch them onto investigate problems.

    TRACE

    Recommendation:Trace level logging should generally stay disabled.