Release Notes

v0.3.0

DefraDB v0.3 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

There are several new features in this release, and we invite you to review the official changelog below. Some highlights are various new features for Grouping & Aggregation for the query system, like top-level aggregation and group filtering. Moreover, a brand new Query Explain system was added to introspect the execution plans created by DefraDB. Lastly we introduced a revamped CLI configuration system.

This release does include a Breaking Change to existing v0.2.x databases. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network/.

Features

• Add named config overrides (#659)
• Expose color and caller log options, add validation (#652)
• Add ability to explain groupNode and it's attribute(s). (#641)
• Add primary directive for schema definitions (@primary) (#650)
• Add support for aggregate filters on inline arrays (#622)
• Add explainable renderLimitNode & hardLimitNode attributes. (#614)
• Add support for top level aggregates (#594)
• Update countNode explanation to be consistent. (#600)
• Add support for stdin as input in CLI (#608)
• Explain cid & field attributes for dagScanNode (#598)
• Add ability to explain dagScanNode attribute(s). (#560)
• Add the ability to send user feedback to the console even when logging to file. (#568)
• Add ability to explain sortNode attribute(s). (#558)
• Add ability to explain sumNode attribute(s). (#559)
• Introduce top-level config package (#389)
• Add ability to explain updateNode attributes. (#514)
• Add typeIndexJoin explainable attributes. (#499)
• Add support to explain countNode attributes. (#504)
• Add CORS capability to HTTP API (#467)
• Add explaination of spans for scanNode. (#492)
• Add ability to Explain the response plan. (#385)
• Add aggregate filter support for groups only (#426)
• Configurable caller option in logger (#416)
• Add Average aggregate support (#383)
• Allow summation of aggregates (#341)
• Add ability to check DefraDB CLI version. (#339)

Fixes

• Add a check to ensure limit is not 0 when evaluating query limit and offset (#706)
• Support multiple --logger flags (#704)
• Return without an error if relation is finalized (#698)
• Logger not correctly applying named config (#696)
• Add content-type media type parsing (#678)
• Remove portSyncLock deadlock condition (#671)
• Silence cobra default errors and usage printing (#668)
• Add stdout validation when setting logging output path (#666)
• Consider --logoutput CLI flag properly (#645)
• Handle errors and responses in CLI client commands (#579)
• Rename aggregate gql types (#638)
• Error when attempting to insert value into relationship field (#632)
• Allow adding of new schema to database (#635)
• Correctly parse dockey in broadcast log event. (#631)
• Increase system's open files limit in integration tests (#627)
• Avoid populating order.ordering with empties. (#618)
• Change to supporting of non-null inline arrays (#609)
• Assert fields exist in collection before saving to them (#604)
• CLI init command to reinitialize only config file (#603)
• Add config and registry clearing to TestLogWritesMessagesToFeedbackLog (#596)
• Change $eq to _eq in the failing test. (#576)
• Resolve failing HTTP API tests via cleanup (#557)
• Ensure Makefile compatibility with macOS (#527)
• Separate out iotas in their own blocks. (#464)
• Use x/cases for titling instead of strings to handle deprecation (#457)
• Handle limit and offset in sub groups (#440)
• Issue preventing DB from restarting with no records (#437)
• log serving HTTP API before goroutine blocks (#358)

Testing

• Add integration testing for P2P. (#655)
• Fix formatting of tests with no extra brackets (#643)
• Add tests for averageNode explain. (#639)
• Add schema integration tests (#628)
• Add tests for default properties (#611)
• Specify which collection to update in test framework (#601)
• Add tests for grouping by undefined value (#543)
• Add test for querying undefined field (#544)
• Expand commit query tests (#541)
• Add cid (time-travel) query tests (#539)
• Restructure and expand filter tests (#512)
• Basic unit testing of node package (#503)
• Test filter in filter tests (#473)
• Add test for deletion of records in a relationship (#329)
• Benchmark transaction iteration (#289)

Refactoring

• Improve CLI error handling and fix small issues (#649)
• Add top-level version package (#583)
• Remove extra log levels (#634)
• Change sortNode to orderNode. (#591)
• Rework update and delete node to remove secondary planner (#571)
• Trim imported connor package (#530)
• Internal doc restructure (#471)
• Copy-paste connor fork into repo (#567)
• Add safety to the tests, add ability to catch stderr logs and add output path validation (#552)
• Change handler functions implementation and response formatting (#498)
• Improve the HTTP API implementation (#382)
• Use new logger in net/api (#420)
• Rename NewCidV1_SHA2_256 to mixedCaps (#415)
• Remove utils package (#397)
• Rework planNode Next and Value(s) function (#374)
• Restructure aggregate query syntax (#373)
• Remove dead code from client package and document remaining (#356)
• Restructure datastore keys (#316)
• Add commits lost during github outage (#303)
• Move public members out of core and base packages (#295)
• Make db stuff internal/private (#291)
• Rework client.DB to ensure interface contains only public types (#277)
• Remove GetPrimaryIndexDocKey from collection interface (#279)
• Remove DataStoreKey from (public) dockey struct (#278)
• Renormalize to ensure consistent file line termination. (#226)
• Strongly typed key refactor (#17)

Documentation

• Use permanent link to BSL license document (#692)
• README update v0.3.0 (#646)
• Improve code documentation (#533)
• Add CONTRIBUTING.md (#531)
• Add package level docs for logging lib (#338)

Tooling

• Include all touched packages in code coverage (#673)
• Use gotestsum over go test (#619)
• Update Github pull request template (#524)
• Fix the cross-build script (#460)
• Add test coverage html output (#466)
• Add linter rule for goconst. (#398)
• Add github PR template. (#394)
• Disable auto-fixing linter issues by default (#429)
• Fix linting of empty else code blocks (#402)
• Add the gofmt linter rule. (#405)
• Cleanup linter config file (#400)
• Add linter rule for copyright headers (#360)
• Organize our config files and tooling. (#336)
• Limit line length to 100 characters (linter check) (#224)
• Ignore db/tests folder and the bench marks. (#280)

Continuous Integration

• Fix circleci cache permission errors. (#371)
• Ban extra elses (#366)
• Fix change-detection to not fail when new tests are added. (#333)
• Update golang-ci linter and explicit go-setup to use v1.17 (#331)
• Comment the benchmarking result comparison to the PR (#305)
• Add benchmark performance comparisons (#232)
• Add caching / storing of bench report on default branch (#290)
• Ensure full-benchmarks are ran on a PR-merge. (#282)
• Add ability to control benchmarks by PR labels. (#267)

Chore

• Update APL to refer to D2 Foundation (#711)
• Update gitignore to include cmd folders (#617)
• Enable random execution order of tests (#554)
• Enable linters exportloopref, nolintlint, whitespace (#535)
• Add utility for generation of man pages (#493)
• Add Dockerfile (#517)
• Enable errorlint linter (#520)
• Binaries incmd folder, examples in examples folder (#501)
• Improve log outputs (#506)
• Move testing to top-level tests folder (#446)
• Update dependencies (#450)
• Update go-ipfs-blockstore and ipfs-lite (#436)
• Update libp2p dependency to v0.19 (#424)
• Update ioutil package to io / os packages. (#376)
• git ignore vscode (#343)
• Updated README.md contributors section (#292)
• Update changelog v0.2.1 (#252)

v0.2.1

This is a minor release that resolves several bugs and implements some minor features that are only extensions of features released in v0.2.

Features

• Add ability to delete multiple documents using filter (#206)
• Add ability to delete multiple documents, using multiple ids (#196)

Fixes

• Concurrency control of Document using RWMutex (#213)
• Only log errors and above when benchmarking (#261)
• Handle proper type conversion on sort nodes (#228)
• Return empty array if no values found (#223)
• Close fetcher on error (#210)
• Installing binary using defradb name (#190)

Tooling

• Add short benchmark runner option (#263)

Documentation

• Add data format changes documentation folder (#89)
• Correcting typos (#143)
• Update generated CLI docs (#208)
• Updated readme with P2P section (#220)
• Update old or missing license headers (#205)
• Update git-chglog config and template (#195)

Refactoring

• Introduction of logging system (#67)
• Restructure db/txn/multistore structures (#199)
• Initialize database in constructor (#211)
• Purge all println and ban it (#253)

Testing

• Detect and force breaking filesystem changes to be documented (#89)
• Boost collection test coverage (#183)

Continuous integration

• Combine the Lint and Benchmark workflows so that the benchmark job depends on the lint job in one workflow (#209)
• Add rule to only run benchmark if other check are successful (#194)
• Increase linter timeout (#230)

Chore

• Remove commented out code (#238)
• Remove dead code from multi node (#186)

v0.2.0

DefraDB v0.2 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

This release is jam-packed with new features and a small number of breaking changes. Read the full changelog for a detailed description. Most notable features include a new Peer-to-Peer (P2P) data synchronization system, an expanded query system to support GroupBy & Aggregate operations, and lastly TimeTraveling queries allowing to query previous states of a document.

Much more than just that has been added to ensure we're building reliable software expected of any database, such as expanded test & benchmark suites, automated bug detection, performance gains, and more.

This release does include a Breaking Change to existing v0.1 databases regarding the internal data model, which affects the "Content Identifiers" we use to generate DocKeys and VersionIDs. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network.

Features

• Added Peer-to-Peer networking data synchronization (#177)
• TimeTraveling (History Traversing) query engine and doc fetcher (#59)
• Add Document Deletion with a Key (#150)
• Add support for sum aggregate (#121)
• Add support for lwwr scalar arrays (full replace on update) (#115)
• Add count aggregate support (#102)
• Add support for named relationships (#108)
• Add multi doc key lookup support (#76)
• Add basic group by functionality (#43)
• Update datastore packages to allow use of context (#48)

Bug fixes

• Only add join if aggregating child object collection (#188)
• Handle errors generated during input object thunks (#123)
• Remove new types from in-memory cache on generate error (#122)
• Support relationships where both fields have the same name (#109)
• Handle errors generated in fields thunk (#66)
• Ensure OperationDefinition case has at least one selection(#24)
• Close datastore iterator on scan close (#56) (resulted in a panic when using limit)
• Close superseded iterators before orphaning (#56) (fixes a panic in the join code)
• Move discard to after error check (#88) (did result in panic if transaction creation fails)
• Check for nil iterator before closing document fetcher (#108)

Tooling

• Added benchmark suite (#160)

Documentation

• Correcting comment typos (#142)
• Correcting README typos (#140)

Testing

• Add transaction integration tests (#175)
• Allow running of tests using badger-file as well as IM options (#128)
• Add test datastore selection support (#88)

Refactoring

• Datatype modification protection (#138)
• Cleanup Linter Complaints and Setup Makefile (#63)
• Rework document rendering to avoid data duplication and mutation (#68)
• Remove dependency on concrete datastore implementations from db package (#51)
• Remove all errors.Wrap and update them with fmt.Errorf. (#41)
• Restructure integration tests to provide better visibility (#15)
• Remove schemaless code branches (#23)

Performance

• Add badger multi scan support (#85)
• Add support for range spans (#86)

Continous integration

• Use more accurate test coverage. (#134)
• Disable Codecov's Patch Check
• Make codcov less strict for now to unblock development (#125)
• Add codecov config file. (#118)
• Add workflow that runs a job on AWS EC2 instance. (#110)
• Add Code Test Coverage with CodeCov (#116)
• Integrate GitHub Action for golangci-lint Annotations (#106)
• Add Linter Check to CircleCi (#92)

Chore

• Remove the S1038 rule of the gosimple linter. (#129)
• Update to badger v3, and use badger as default in memory store (#56)
• Make Cid versions consistent (#57)