v0.9.0
 · 阅读需 11 分钟
Release date: July 16, 2024
👍 Highlights
- Log Engine: GreptimeDB is now a unified time-series database for both metrics, logs, and events (trace in plan).
 - Remote WAL is significantly improved and is now recommended to turn on.
 - Table View: You can now 
CREATE VIEWon tables and treat them as logical table. - Short interval literal for user experience.
 - InfluxDB Merge Read: You can now use the 
merge_mode=last_non_nulltable option to allow SELECT result merges all the last non-null fields. - gRPC TLS: You can now set up the 
grpc.tlsconfig to turn on TLS for gRPC connection. - Manually Compact in Strict Window: You can now use 
SELECT COMPACT_TABLE("monitor", "strict_window", "3600");to manually schedule a compaction task, splitting SST files strictly into time windows. The third option is the time window's width in seconds. 
Breaking changes
- refactor!: unify 
FrontendOptionsandDatanodeOptionsby usingGrpcOptionsby @realtaobo in #4088 - feat!: reduce sorted runs during compaction by @v0y4g3r in #3702
 - fix!: forbid to change information_schema by @CookiePieWw in #4233
 - feat!: remove alias 
metasrv-addrby @discord9 in #4239 - feat!: Set merge mode while creating table in influx handler by @evenyag in #4299
 
🚀 Features
- feat: log ingestion support by @paomian in #4014
 - feat(pipeline): transform support on_failure by @yuanbohan in #4123
 - feat(pipeline): gsub processor by @yuanbohan in #4121
 - feat: enable metasrv to accept flownode's heartbeats by @fengjiachun in #4160
 - feat: update default size of bgworkers, add hbworkers by @sunng87 in #4129
 - feat: implement the 
OrderedBatchProducerby @WenyXu in #4134 - feat(flow): support datafusion scalar function by @discord9 in #4142
 - feat: Implement memtable range by @evenyag in #4162
 - feat: update dashboard to v0.5.1 by @ZonaHex in #4171
 - feat: register flow node by @fengjiachun in #4166
 - feat: show database options by @killme2008 in #4174
 - feat: use the write runtime to handle the heartbeats by @fengjiachun in #4177
 - feat: PREPARE and EXECUTE statement from mysql client by @CookiePieWw in #4125
 - feat: introduce chaos crds by @WenyXu in #4173
 - feat: add num_rows and num_row_groups to manifest by @waynexia in #4183
 - feat: make RegionScanner aware of PartitionRange by @waynexia in #4170
 - feat: update dashboard to v0.5.2 by @ZonaHex in #4185
 - feat(puffin): support lz4 compression for footer by @zhongzc in #4194
 - feat: heartbeat task&peer lookup in proc by @discord9 in #4179
 - feat(puffin): introduce puffin manager trait by @zhongzc in #4195
 - feat(sql): improve interval expression, support shortened version by @etolbakov in #4182
 - feat: Dedup strategy that keeps the last not null field by @evenyag in #4184
 - feat(puffin): implement CachedPuffinWriter by @zhongzc in #4203
 - feat: FlownodeClient by @discord9 in #4206
 - feat: bulk memtable codec by @v0y4g3r in #4163
 - feat(puffin): implement CachedPuffinReader by @zhongzc in #4209
 - feat: validate partition rule on create table by @waynexia in #4213
 - feat(servers): make http timeout and body limit optional by @fengjiachun in #4217
 - feat(sql): add casting support for shortened intervals by @etolbakov in #4220
 - feat: Implements 
merge_moderegion options by @evenyag in #4208 - feat: output multiple partition in MergeScanExec by @waynexia in #4227
 - feat: add build info for flow heartbeat task by @fengjiachun in #4228
 - feat: register & deregister region failure detectors actively by @WenyXu in #4223
 - feat: forgot collect flownode clusterinfo handler by @fengjiachun in #4236
 - feat: basic implement of 
matchesfn by @waynexia in #4222 - feat: flow cli for distributed by @discord9 in #4226
 - feat(puffin): implement MokaCacheManager by @zhongzc in #4211
 - feat(puffin): complete dir support by @zhongzc in #4240
 - feat: introduce the interface of 
RemoteJobSchedulerby @zyy17 in #4181 - feat(fulltext_index): introduce creator by @zhongzc in #4249
 - feat: dbeaver mysql compatibility, use statement and information_schema.tables by @sunng87 in #4218
 - feat: provide a simple way to create metaclient by @fengjiachun in #4257
 - feat: implement naive fuzz test for region migration by @WenyXu in #4252
 - feat: update dashboard to v0.5.3 by @ZonaHex in #4262
 - feat: introduce 
FlowRouteValueby @WenyXu in #4263 - feat(fulltext_index): integrate puffin manager with inverted index applier by @zhongzc in #4266
 - feat: handle AND/OR and priority in matches fn by @waynexia in #4270
 - feat: store peer info in 
TableFlowValueby @WenyXu in #4280 - feat(fuzz): enhance condition check of region migration finish by @WenyXu in #4283
 - feat: delete pipeline by @shuiyisong in #4156
 - feat: add path prefix label to object storage metrics by @sunng87 in #4277
 - feat: add naive region failover test for metric table by @WenyXu in #4269
 - feat: expose merge_mode option by @evenyag in #4289
 - feat: refine scan metrics logging by @evenyag in #4296
 - feat: make flow distributed work&tests by @discord9 in #4256
 - feat(remote wal): set default compression to LZ4 by @WenyXu in #4294
 - feat(sql): add iso-8601 format support for intervals by @etolbakov in #4291
 - feat(fulltext_index): integrate full-text indexer with sst writer by @zhongzc in #4302
 - feat: impl show table status by @killme2008 in #4303
 - feat: support inserting into binary value through string by @CookiePieWw in #4197
 - feat(inverted_index): inverted index cache by @v0y4g3r in #4309
 - feat(fulltext_index): allow enable full-text index in SQL and gRPC way by @zhongzc in #4310
 - feat: handle parentheses with unary ops by @waynexia in #4290
 - feat: enhanced the retry logic by adding a random noise by @WenyXu in #4320
 - feat: flownode use 
Inserterto write to database by @discord9 in #4323 - feat: add 
TimeSeriesRowSelectorhint by @evenyag in #4327 - feat: show create view and creating view with columns by @killme2008 in #4086
 - feat: customize copy to parquet parameter by @v0y4g3r in #4328
 - feat: impl drop view by @killme2008 in #4231
 - feat: support 
text/plainformat for log ingestion by @shuiyisong in #4300 - feat: impl optimizer rule to handle last_value case by @waynexia in #4357
 - feat: Implement reader that returns the last row of each series by @evenyag in #4354
 - feat(index): distinguish different types of index metrics by @zhongzc in #4337
 - feat: adding information_schema.views table by @lyang24 in #4342
 - feat: support show views statement by @lyang24 in #4360
 - feat: tweak error and status codes by @killme2008 in #4359
 - feat(fulltext_index): integrate full-text indexer with parquet reader by @zhongzc in #4348
 - feat: flow perf&fix df func call by @discord9 in #4347
 - feat: add a cache for last value result in row group by @evenyag in #4369
 - feat: add PruneReader for optimized row filtering by @v0y4g3r in #4370
 - feat: introduce 'pg_catalog.pg_type' by @J0HN50N133 in #4332
 - feat: improve datafusion external error and mysql error by @killme2008 in #4362
 - feat: Add caching for last row reader and expose cache manager by @v0y4g3r in #4375
 
🐛 Bug Fixes
- fix(sqlness): catch different format timestamp by @WenyXu in #4149
 - fix: region logical regions after catching up by @WenyXu in #4176
 - fix(flow): fix call df func bug&sqlness test by @discord9 in #4165
 - fix: 
region_peersreturns same region_id for multi logical tables by @realtaobo in #4190 - fix(sql): improve compound signed number processing by @etolbakov in #4200
 - fix: align workflows again for the troublesome GHA by @waynexia in #4196
 - fix: format error correctly by @WenyXu in #4204
 - fix: wrong frontend registration address by @killme2008 in #4199
 - fix: add 
serialize_ignore_column_ids()to fix deserialize region options failed from json string by @zyy17 in #4229 - fix(puffin): fix dependency by @v0y4g3r in #4267
 - fix(fuzz): generate valid string by @WenyXu in #4281
 - fix: enhance ColumnOption::DefaultValue formatting for string values by @WenyXu in #4287
 - fix: enable space string in yaml value by @shuiyisong in #4286
 - fix: align pre-commit config with make file by @shuiyisong in #4292
 - fix: call df_func with literal by @discord9 in #4265
 - fix: prepare inserting with column defaults not work, #4244 by @killme2008 in #4272
 - fix: deregister failure detector in region migration by @WenyXu in #4293
 - fix(ci): remove sqlness state in success by @waynexia in #4313
 - fix: test_fulltext_intm_path by @zhongzc in #4314
 - fix: error on show databases in non-default catalog by @sunng87 in #4316
 - fix: panic while reading information_schema. KEY_COLUMN_USAGE by @killme2008 in #4318
 - fix: support unary operator in default value, partition rule and prepare statement by @waynexia in #4301
 - fix(sqlness): relax start time regex to match various precisions by @waynexia in #4326
 - fix: permission denied is 403 by @sunng87 in #4350
 - fix(config): enable file engine by default by @WenyXu in #4345
 - fix: build info should use build time env var by @tisonkun in #4343
 - fix: remove path label for cache store by @sunng87 in #4336
 - fix: scan hint checks order asc by @evenyag in #4365
 
🚜 Refactor
- refactor: add 
Compactortrait to abstract the compaction by @zyy17 in #4097 - refactor: make region manifest checkpoint ran in background by @MichaelScofield in #4133
 - refactor: Decouple dedup and merge by @evenyag in #4139
 - refactor: make 
RegionOptionsandMergeOutputserializable by @zyy17 in #4180 - refactor: add 
region_dirin CompactionRegion by @zyy17 in #4187 - refactor: migrate region failover implementation to region migration by @WenyXu in #4172
 - refactor: add 
SerializedPickerOutputand field modification ofCompactorRequestby @zyy17 in #4198 - refactor(flow): make 
from_substrait_*async& worker handle refactor by @discord9 in #4210 - refactor: expose 
DatanodeBuilder::build_object_store_manager()andMitoConfig::sanitize()by @zyy17 in #4212 - refactor: use ObjectStoreManagerRef type in open_compaction_region() and add related unit test by @zyy17 in #4238
 - refactor: add interceptor after Influxdb lines are converted to grpc row insert by @MichaelScofield in #4225
 - refactor: add 
RemoteCompactionerror by @zyy17 in #4251 - refactor: change InvertedIndexWriter method signature to offsets to f… by @v0y4g3r in #4250
 - refactor(puffin): adjust generic parameters by @zhongzc in #4279
 - refactor: use rwlock for modifiable session data by @sunng87 in #4232
 - refactor(inverted_index): integrate puffin manager with sst indexer by @zhongzc in #4285
 - refactor: split match arms in prom_expr_to_plan into smaller methods by @waynexia in #4317
 - refactor: do not print error log on PlanQuery error by @sunng87 in #4322
 - refactor: Remove the StandaloneKafkaConfig struct by @irenjj in #4253
 - refactor: LastRowReader to use LastRowSelector by @v0y4g3r in #4374
 
📚 Documentation
- docs: add guide for tsbs benchmark by @waynexia in #4151
 - docs: remove outdated docs by @evenyag in #4205
 - docs: remove cargo test workspace command by @waynexia in #4325
 - docs(config): add enable_region_failover option to configuration by @WenyXu in #4355
 
⚡ Performance
- perf: optimize RecordBatch to HttpOutput conversion by @waynexia in #4178
 - perf(puffin): not to stage uncompressed blob by @zhongzc in #4333
 - perf: fine–tuned plan steps by @waynexia in #4258
 
🧪 Testing
- test: wait until checkpoint finish by @evenyag in #4202
 - test: add e2e test for region failover by @WenyXu in #4188
 - test: replace unstable output of last value test by @evenyag in #4371
 - test: more sleep when flow insert makes it serial by @discord9 in #4373
 - test(flow): ignore flow tests for now by @discord9 in #4377
 
⚙️ Miscellaneous Tasks
- ci: align docs with develop by @waynexia in #4152
 - chore: bump datafusion version to fix 
last_valueregression by @MichaelScofield in #4169 - chore: enhance add pipeline http api return data by @paomian in #4167
 - chore: highlight our committers in CONTRIBUTING.md by @tisonkun in #4189
 - chore: add AUTHOR.md file by @tisonkun in #4241
 - ci: update centos yum source and specify cargo-binstall version by @zyy17 in #4248
 - chore: reduce insertion size of fuzz test by @WenyXu in #4243
 - chore(ci): add timeout (60min) for fuzz tests by @WenyXu in #4255
 - ci: push latest greptimedb nigthtly build image by @zyy17 in #4260
 - chore: add missing s for 
--metasrv-addrby @discord9 in #4278 - ci: retry on error during installing operator by @WenyXu in #4295
 - chore: remove original region failover implementation by @WenyXu in #4237
 - chore: bump OpenDAL to 0.47.2 by @WenyXu in #4297
 - ci: retry on error or timeout during installing operator by @WenyXu in #4308
 - chore: disable TraceLayer on_failure log by @paomian in #4315
 - ci: add flownode in docker compose by @zyy17 in #4306
 - chore: update project slogan by @tisonkun in #4361
 - chore: bump version to v0.9.0 by @evenyag in #4376
 
◀️ Revert
Build
- build(deps): Upgrade OpenDAL to 0.47 by @tisonkun in #4224
 - build(deps): upgrade opendal to 0.47.3 by @tisonkun in #4307
 - build(deps): switch to upstream by @tisonkun in #4319
 
All Contributors
We would like to thank the following contributors from the GreptimeDB community:
@CookiePieWw, @J0HN50N133, @MichaelScofield, @WenyXu, @ZonaHex, @discord9, @etolbakov, @evenyag, @fengjiachun, @irenjj, @killme2008, @lyang24, @paomian, @realtaobo, @shuiyisong, @sunng87, @tisonkun, @v0y4g3r, @waynexia, @yuanbohan, @zhongzc, @zyy17