Updated at: 2022-12-09 03:49:50
After the log data is aggregated and parsed, the system will match the corresponding Log Library according to the Log Type, and after successful matching, the corresponding Index will be created automatically according to the Index Rule and Total Shards. Besides, the Life Cycle and the Migrate Configuration for Hot-Warm Data can be configured in Log Library.
Click Data Source > Library management > Log Library, to create, delete / batch delete, edit, view, import / export the Log Library.
Note: Log Library deletion is not reversible, so please be careful.
► Create Log Library
• Manually Create Log Library
1) Click Data Source > Library management > Log Library > + New to Create Log Library, as follows: 
Field Name Description Restrictive Condition
Log Library The log library name to identify the different log libraries Name must be globally unique.
Tag The log tag corresponding to the log library -
Log Type The type of log data corresponding to the log library Log Type cannot start with ~ ` ! @ $ ^ & ( ) _ = characters, with no uppercase letter.
Index Rule The index creation rule which determines the period of index creation, supports daily/monthly/yearly creation, and generates index name accordingly -
* Total Shards The total storage capacity of the index
Note: If the index capacity reaches the total storage capacity, the index name is added with incremental suffix, starting from 0 by default, e.g. mysql-2020.06-0, mysql-2020.06-1 ...
A single shard has 15-30GB of storage capacity

Note: 
To avoid unreasonable system data volume and index allocation shards, which can result in storage space insufficiency or waste, please make reasonable space planning as needed, according to the index creation time interval. Please refer to parameters of Business Data Volume, Index Rule and Total Shards, as follows: 
Business Data Volume
(per day)
Index Rule Total Shards Storage Space
data≥90GB Type - Year.Month.Date 4 120GB
60GB≤data<90GB Type - Year.Month.Date 3 90GB
30GB≤data<60GB Type - Year.Month.Date 2 60GB
2GB≤data<30GB Type - Year.Month.Date 2 40GB
1GB≤data<2GB Type - Year.Month 1 15GB
500MB≤data<1GB Type - Year.Month 1 15GB
100MB≤data<500MB Type - Year.Month 1 15GB
data<100MB Type - Year 1 15GB
2) The configuration of the rest parameters in Template Configuration is as follows:
Field Name Description Restrictive Condition
Hot Data Copy The number of hot data backups for the index of the Log Library The minimum value is 0
Life Cycle The retention period for the index of the Log Library -
Fulltext Index The fulltext index status, including 2 types of status: Enable/Disable
Note: Enabling Fulltext Index will increase storage costs and feedback delay in searching for processing results.
-
Field Setting It is to set the Field Name and Field Type of the parsed field Metadata field cannot be modified.
3) The Migrate Configuration for Hot-Warm Data is as follows: 
Field Name Description Restrictive Condition
Migrate Policy It is to set the range of data to be migrated from hot data to warm data, e.g. hot data 3 months ago will be migrated to warm data by default. If by month, the default time is 30 days per month.
Warm Data Copy It is to set the number of backups to migrate from hot data to warm data. The number of copies ranges from 0 to 3, and 0 by default.
4) Click Save to complete the creation of Log Library.
• Automatically Create Log Library
If the Log Library in corresponding log type cannot be found during the index scheduled detection process, the system will automatically create a new Log Library corresponding to that index Log Type. The following is the default parameters for automatic Log Library creation: 
Parameter Name Description
Log Library Data Type + Log Library, for example, if the Data Type is MySQL, the name of the generated Log Library will be mysql Log Library.
Index Rule Type - Year.Month
Total Shards 3
Capacity of Shards 30GB
Hot Data Copy 1
Life Cycle 999 Years
Fulltext Retrieval Enable
Migrate Policy Hot data will be migrated to warm data 3 months later.
Warm Data Copy 1
► Edit/Delete Log Library
Click Data Source > Library management > Log Library, select the target log library and click Edit/Delete to edit or delete the selected log library.

Note: 
1. Automatically created log libraries cannot be edited;
2. If the log library is bound to a non-null index, the log type configuration of the log library cannot be modified;
3. If the log library is bound to a non-null index, the log library cannot be modified.

► Check Log Library
Click Data Source > Log Management > Log Library, select the target log library, click Check in the action column to check Log Library Details, where you can view the Basics, Template Configuration, and configuration parameters for hot data and warm data, as follows: 

► Import/Export Log Library
• Export a log library: In the log library list, tick the log library to export and click the 
icon to export the log library (.json) file;
• Import a log library: In the log library list, click the
icon to select the log library (.json) file in the window, and click Open to import.
► Log Library Sorting
The log library list can be sorted by the fields: Log Library, Current Storage and Create Time.