EthereumJS Documentation

This guide aims to be a both comprehensive and lightweight guide to the EthereumJS ecosystem. It is meant to serve as an internal reference, give guidance for new contributors and overall provide transparency on current and future work and standards and best practices applied.

Introduction

Overview

EthereumJS is the JavaScript / TypeScript project within the Ethereum Foundation. Work is mainly done within the following GitHub organization:

Our central repository is the ethereumjs-vm monorepo hosting a full featured TypeScript Ethereum VM implementation as well as related packages like:

  • Structural blockchain component representations like e.g. the block or tx packages
  • The common package providing central access to chain and hardfork settings
  • An Ethash implementation

Other noteworthy libraries out of the monorepo scope are e.g. an implementation of the Merkle Patricia Tree, the RLP serialization library or a widely used Util library providing utility functions for things like hashing, signatures as well as helpers for address and account management.

Have a look at the organizational GitHub overview page linked above to get an impression what is currently being worked on as well as the libraries available.

Team and Contact

EthereumJS is a strongly community-driven project and the active team is regarded as the sum of people actively contributing to the libraries, ongoing development is ensured by the JavaScript team from the Ethereum Foundation.

For technical questions and getting in touch you can use our Discord server:

Organizational questions are centered and discussed on the organization repo:

Ongoing Work Tasks

The following is an overview on ongoing work tasks to get an idea on the current focus of work. This is also serving internal accounting purposes.

Note

This list is focussing on reoccuring work tasks, for an overview on dedicated new projects have a look at the Roadmap section.

W1 - Virtual Machine Development

One strong emphasis of EthereumJS work is on maintaining and further developing a robust and up-to-date JavaScript virtual machine implementation (ethereumjs-vm).

Main tasks around this are:

  • Updating the VM on new hardforks
  • Targeting compliance with the latest consensus test suite releases
  • Implementing feature requests from the community (Truffle, Remix, others), e.g. to provide better debugging functionalities
  • Ongoing refactoring work to open up new use cases

W2 - Library Modernization

EthereumJS libraries provide robust and solid implementations surving the dedicated purposes. Along there is an ongoing effort to integrate new Javascript respetively TypeScript language feature and adopt to new coding practices to provide the community with secure and easy-to-develop upon libraries and APIs.

Some things done lately in this regard:

  • Using ES6 classes for structuring library components
  • JavaScript Promise based interfaces (in contrast to callback logic)
  • TypeScript transition of all major libraries
  • Updating on security improving language features (block-scoped variables,…)
  • Improving on code readability (destructuring of objects,…)

W3 - Bug Fixes and Maintenance

EthereumJS libraries are widely used in production - often in security-sensitive contexts - and there is an ongoing effort to keep libraries up-to-date and secure.

Main tasks around this:

  • Fix bugs reported by the community in a timely fashion
  • Keep library dependencies up-to-date
  • Adopt libraries to various user work environments and build pipelines (browser, React,…)
  • Be responsive to feature requests from the community

W4 - Testing and CI

To provide a high level of reliable we target a high test coverage on all of our libraries and writing new tests and integrate these in the everyday work process (CI) is an ongoing effort.

Efforts include:

  • Improve test coverage for library APIs
  • Add and maintain integration tests (with a focus on browser testing)
  • Integrate test runs / coverage reports into CI process
  • Benchmark libraries, performance improvements for both library execution and tests

W5 - Community Work

There is a high level of engagement from the community with the different EthereumJS libraries and there are countless examples for both evolutionary updates as well as high-quality and broadly scoped feature contributions from the community.

We are determined to put substantial ressources here to further support exchange with and engagement from the community.

Related tasks are:

  • Help onboard new contributors, give introductory guidance
  • Review of Pull Requests
  • Accompany community development work
  • Management and structuring of issues and PRs
  • Responsiveness on communication channels

W6 - Accessibility

Very much related to the community efforts (W5) is the goal of making libraries generally as easily approachable as possible and so to lower the barrier to engage and minimize the need to to do one-to-one explanations on how things work.

Tasks include:

  • Provide up-to-date and consistent API documentation
  • Instructions on environment setup and installation, developer docs
  • Easy to recreate and up-to-date examples in README
  • Common standards and standard documentation (these docs :-)) whenever possible
  • Easy to understand, modular and documented source code

Contributing

Everyone is invited to contribute to the EthereumJS libraries (see also our Code of Conduct). These are some guidelines to help you get started!

Where to Contribute

Picking up some Issues

There are labelled issues on all our libraries, see e.g. the issue pages of the VM or the Merkle Tree libraries, sorting issues on things like effort needed, priority or type.

Feel free to pick any issue you think is suitable for you to work on, then you might also want to drop a note on the issue page that you are working on the issue.

Some issues are also labelled with help wanted and/or good first issue, indicating that they are in particular suitable to get started.

Some generic Tasks

There are also various generic tasks which constantly needs help, you can also have a look at the Ongoing Work Tasks section to get an overview here.

Many of these things are not listed as issues, but are nevertheless a good place to start especially for new contributors. This includes:

All these things are a good way to gently get in touch with the inner workings of a library without directly have to manipulate production code directly.

How to Start

Introductory Information

Once you have chosen what you want to work on you can actually grab your coffee, take your laptop to a quiet place to work and start hacking! 😄

Have a look at the Git Guidelines and the Workflow Best Practices sections for some Git and overall work instructions being common practice within the EthereumJS ecosystem.

The Technical Reference chapter generally contains some overview information on the development environment, programming language and tools used throughout the libraries. Have a broader look on what is relevant for you to successfully work on your selected task.

Get in Touch

Generally: just get in touch. Early on - see Team and Contact section. Feel free to ask everything you need to know, there is no question which shouldn’t been asked and there will likely be someone who can give you some guidance along the way.

Technical Reference

This guide gives an overview on common practices and technical standards shared within the EthereumJS ecosystem.

Development

Node.js

Development Version

Runtime environment for development is node.js.

Development should always be possible running latest LTS Node.js version, see Node.js release schedule table.

Node Version Status Latest Status Change
Node.js 12 Supported 2020-10-19
package-lock Files

The usage of package-lock files has been discussed extensively within the EthereumJS community and team, see e.g. this thread for some background on discussions which took place.

Latest policy agreement here is that package-locks are not regarded as strictly necessary for the libraries but are recently under reconsideration due to other benefits (speed, caching, reliability, reproducibility, etc.) and using a lockfile is taken on a case-by-case basis.

JavaScript

All libraries are transpiled to a lower common denominator JavaScript version (see section below), this section describes what language features are agreed upon to be be used in the non-transpiled source code of the libraries.

Supported Versions

Features from the following JavaScript version standards are safe to be used:

Partially Supported Versions

Be careful when using language features from the following JavaScript standards:

Feature Notes

BigInt

BigInt support is an often requested feature within the EthereumJS ecosystem and we are constantly re-evaluating usage. Current discussion state is that we are not quite there on the browser/runtime support side yet to integrate in the libraries, see e.g. Can I use bigint? page for context.

We are getting close though, so if you feel a pressing need here it might be worth to re-trigger the discussion.

TypeScript

All the major EthereumJS libraries use TypeScript,

TypeScript version and configuration is centrally managed in the ethereumjs-config typescript package.

Linting and Formatting

Linting and formatting of package source code can be triggered on the different libraries with an npm run lint respectively a npm run lint:fix command from package.json.

Tool usage and configuration is centrally managed in the ethereumjs-config lint package.

Distribution

Transpilation Targets

For TypeScript libraries, transpilation is done through the TypeScript compiler tsc command line tool.

See the ethereumjs-config typescript tsconfig.*.json files for an overview on transpilation targets.

Node.js Distribution Versions

The following table gives an overview on the targeted Node.js version support:

Node Version Status Latest Status Change
Node.js 4 Dropped 2018-10-01
Node.js 6 Dropped 2019-02-19
Node.js 8 Dropped 2020-01-29
Node.js 10 Supported 2020-03-01
Node.js 12 Supported 2019-06-01
Node.js 13 Partly Supported 2020-10-19
Node.js 14 Mostly Untested 2020-10-19

For a concrete overview on supported Node.js versions have a look at the GitHub Actions CI setup within the .github folder of a repository, see build.yml as an example from the merkle-patricia-tree library.

Browser Compatibility

Most libraries are tested with Karma for browser compatibility, see karma.conf.js from the merkle-patricia-tree library for an example setup.

Releases

Releases on libraries follow Semantic Versioning, normally releases are published on npm and as a tagged release on GitHub in the Releases section.

Every library contains a CHANGELOG.md file in the root directory, listing the changes on the respective release versions (see e.g. CHANGELOG.md of the ethereumjs-util library), the changelog entry is copied to the GitHub release section on publication of a new release.

Releases go through a PR (see example PR <https://github.com/ethereumjs/ethereumjs-util/pull/155/files> on ethereumjs-util v6.0.0 release), containing the package.json version number update, a new CHANGELOG entry and eventually some update on the docs.

Git Workflow

Branching Model

We are using a feature-centric branching model, the GitHub flow model is coming very much close.

Development of new features is taking place on a dedicated branch and should have some descriptive name for the work done (e.g. api-doc-fixes, remove-vm-accesses-to-statemanager-trie-cache, new-bloom-filter-tests).

Once work on the feature branch is completed and all tests and checks from CI (see Continuous Integration (CI)) pass it goes through a review and eventually discussion process and is afterwards merged into a protected master branch. The master branch should always be stable and theoretically ready for deployment.

Git Guidelines

Some guidelines for the EthereumJS libraries when working with Git version control:

Feature Branch for All PRs

Always do your work on a separate feature branch (see Branching Model), this also applies when doing work from an own fork of a library.

This makes it easier for reviewers and others interested to test your code locally by fetching your code changes from your remote feature branch.

Separate PRs for Separate Features

If you have separate things you want to change on a library, do separate PRs for this. So if you e.g. have some ideas for how to improve the build process and want to fix some bug from an issue, theses are two separate PRs.

This is a precondition for a successful review of a PR, since a reviewer has a smaller subset of changes and can connect changes definitively to a certain feature. It also avoids the situation where unexpected discussions and disagreements on a certain subfeature set blocks the whole PR with all other changes.

Meaningful Commit History

Make sure that you end up with a meaningful commit history on your work:

  • Choose self-descriptive commit messages
  • Avoid inconsistent state between commits
  • If you do changes correcting your prior committed work, rebase and squash commits afterwards

Note

Rebasing can be a hairy process, if you do for the first time it is highly recommended to do a local backup of your repository.

Note

Rebase work like the above can normally be done with git rebase -i master from the feature branch with an up-to-date master branch.

Regular Master Rebase

PRs are only reviewed if the branch is up-to-date on the latest master changes. Rebase your branch often (with git rebase master) and force-push the changes, to make sure that your changes work well on top of the latest commits and tests keep passing.

Workflow Best Practices

Some best practices which turned out to be practical over time and should be followed when working on a new feature:

In doubt: Issue before PR

If you are planning on introducing major feature changes on a library file an issue and describe what you are up to before directly work on a PR. This gives others the chance to discuss around your intended changes and avoids potential further conflicts along the road.

This especially applies for stuff like:

  • Introducing new language features (Promises,…)
  • Changing the API of a library
  • Planning security-sensitive changes
  • Switch or introduce new tooling
Describe your Work

Take some time to make both the scope of your work and your work process transparent for others. This will ease both discussions and the review process around the work being done.

In particular:

  • Do a proper and complete task description on your issue or PR
  • Give some regular updates on the current status of your work
  • Especially: drop a note once you are ready

Pull Request Reviews

All PRs making changes to the production code base are going through a review process. This will normally take some time and will come along with some back-and-forth between contributor and reviewer until everyone is happy.

Code Quality

Testing

Test Framework

Most EthereumJS libraries use tape for running tests. Have a look at one of the libraries (e.g. merkle-patricia-tree) for reference.

Code Coverage

For coverage runs nyc is used. Results are passed on to the coveralls.io service for coverage reports on CI runs.

Tool usage and configuration is centrally managed in the ethereumjs-config coverage package.

Documentation

Libraries come with an API documentation generated automatically from comments in the code.

To generate API documentation for a TypeScript project, TypeDoc is employed. By default, TypeDoc generates HTML documentation. In order to generate Markdown suitable for GitHub, the typedoc-plugin-markdown can be used as a theme for TypeDoc.

Apart from that, the following documentation should be kept up-to-date:

  • README with setup and installation instructions
  • Usage instructions, up-to-date code examples

Continuous Integration (CI)

All EthereumJS libraries use GitHub Actions <https://github.com/features/actions> for CI runs on every PR submitted. Have a look at the files in the .github/workflows folder from a repository to get an overview on what is run during the CI process.

Security

Security aspects around the EthereumJS libraries should be taken seriously, since many of the libraries are used in production in security-sensitive environments.

Dependency Management

Dependencies are a main source for also importing security vulnerabilities on a library, so the set of dependencies on the libraries should be actively managed and regularly reviewed.

Some guidelines:

Minimal Dependencies

Every introduction of a new dependency on a library should be carefully considered and there has to be solid argument why a new dependency is necessary. This primarily applies for production but also for development dependencies. Dependencies listed in package.json should be reviewed on a regular basis if they are still necessary or could be removed.

Established and maintained Dependencies

Only (somewhat) established and actively maintained dependencies should be used on the libraries. Some indicators for a not-so-established dependency:

  • Low number of GitHub stars or a similar metric
  • No commit activity for a longer period of time
  • Low download rate on npm
Regular Dependency Updates

Dependency versions should be updated on a regular basis, this is also very welcome to be done as a first-time-contributor PR. Don’t underestimate this task though, since a dependency update almost always come along with some necessary changes on a library. It is recommended to always only do one dependency at a time, since it becomes easier to attribute if things break at some point.

Shared Library Resources

The following libraries set up some shared infrastructure for certain purposes.

ethereumjs-testing

The ethereumjs-testing library is a proxy library for the common Ethereum Tests consensus tests.

The common test library is integrated as a submodule and there are tagged releases (no publishing to npm due to size constraints) which can be used for running the latest tests in JavaScript libraries.

ethereumjs-config

The ethereumjs-config library provides a set of unified configuration options (e.g. on the TypeScript configuration or on the linting setup) for the various EthereumJS libraries.

Roadmap

Active Projects

Note

These docs are currently not used to track active projects. Please have a look at the EthereumJS Organization repository issues to get an impression what is currently being discussed and worked on.

R18-2 EthereumJS Client

Note

Project description and milestones of this project are not up-to-date.

Although popular clients like Geth and Parity already exist, given the popularity of the JavaScript language we have finally started the development of a dedicated JavaScript Ethereum client with fast- and light-sync support. Development started in June 2018 on https://github.com/ethereumjs/ethereumjs-client and reactions from the community have been extremely positive.

Initially, rather than focus on building a consensus-critical client, we want to focus on the following use cases (in order of importance):

  • In-Browser/NodeJS research & development (sharding, libp2p, etc.) mainly supported by a modular and extensible (plugin-based) architecture
  • In-Browser education applications
  • In-Browser/NodeJS client simulations and visualizations
  • In-Browser light client (Metamask without Infura)

Generally the EthereumJS client project has larger similarities with the scope of the Trinity project of the Python team. Since JavaScript (like Python) is an extremely popular and widely used language, this will draw in a whole new class of developers who were not able to experiment with and develop on Ethereum client technologies before.

One side goal being nevertheless important is finally to use the client development as a proxy to “harden” the other EthereumJS libraries against a real production environment and serve as a better foundation for testing for our Virtual Machine implementation.

The client project will also build a solid foundation for continued internal research and development efforts. We’ve already incorporated support for libp2p as an alternate transport to RLPx/Devp2p that enables the in-browser light client use-case. In the future, we hope to implement a Clique PoA engine and test it on the Goerli testnet, and later build a working Ethereum 2.0 stateless client.

At a later point it is also be desired to have a dedicated website for the client (similar to https://geth.ethereum.org/) to have a more visible entry point and source for information around the client for the community.

Timeline
  • Q3 2018
    • ✅ Proof-of-concept chain sync (fast and light)
    • ✅ Libp2p networking and browser support
  • Q4 2018
    • ✅ Achieve > 90% code coverage via unit/integration tests
  • Q1 2019
    • ✅️ Reliable mainnet chain sync (fast and light)
  • Q2 2019
    • 🛠 Block validation
    • 🛠 Implement state downloading
  • Q3 2019
    • ⭕ Test setup on hive
    • ⭕ Determine Ethereum 2.0 strategy (ShasperJS collaboration? stateless client?)
  • Q4 2019
    • ⭕ Alpha release of client

Considered Projects

Projects currently under consideration or in a draft state.

R19-2 AssemblyScript (eWASM)

Currently the eWASM team is working on the implementation of an upgraded Ethereum virtual machine (VM), replacing the existing EVM with a WebAssembly (WASM) compatible VM, a testnet supporting this is already up and running.

This will allow to write smart contracts in various classical non-blockchain specific languages. One language specifically targeted for support by the eWASM team is AssemblyScript. This language is a subset of TypeScript which is basically JavaScript with type additions. TypeScript is already supported and will become the default language for EthereumJS libraries once R18-1 Transition to TypeScript is completed.

While AssemblyScript is syntactically compatible with (e)WASM it will nevertheless take some signifcant high-level work to make this a trusted Ethereum smart contract language.

Tasks in this regard are:

  • Define and spec out some practically usable high-level API
  • Create code examples
  • Build up some tooling infrastructure
  • Create helper libraries
  • Think about security best practices
  • …

It would be some natural fit for the EthereumJS team to take on the high-level part of the AssemblyScript work (in contrast to the low-level task to secure AssemblyScript to eWASM compatibility) due to the familiarity with the language and the close relationship with the eWASM team.

Finished Projects

R18-1 Transition to TypeScript

There is currently a transition of EthereumJS libraries from JavaScript to TypeScript in the works. This is a somewhat larger effort since it not only requires significant updates to the source code but also comes with changes to the toolchain (e.g. regarding testing, code analysis (linting) and formatting) on all libraries transitioned. This fact nevertheless is an opportunity to rethink parts of tooling and systematically introduce improved procedures along the way.

Bringing type safety to the EthereumJS libraries should bring large mid-term benefits regarding overall security and robustness of the libraries.

Timeline
  • November 2018
    • ✅ Ad-hoc team, tooling discussion, kick-off at Devcon4
  • December 2018
  • February 2019
    • ✅ Three+ more completed transitions (acount, util, common)
    • ✅ Stable toolchain, ethereumjs-config v1.1.0 release
    • ✅ TypeScript preparation for VM, merkle-patricia-tree library (code modernization, ES6)
  • April 2019 - ✅ blockchain library TypeScript release
  • August 2019 - ✅ VM TypeScript release - ✅ All major transitions completed including VM, merkle-patricia-tree

Stalled Projects

R19-3 eWASM VM/Refactoring

eWASM is being seriously considered as an alternative for the current Ethereum Virtual Machine (EVM) for Ethereum 2.0. Parts of eWASM might get integrated into the current Ethereum mainnet as part of the Eth 1.x roadmap.

As such, a major focus of the team is to provide tooling for eWASM and integrate it into the Javascript VM implementation to facilitate eWASM research and prepare for when eWASM makes it to mainnet.

The current implementation is tightly coupled to EVM as the only VM. Therefore part of this project is to refactor parts of ethereumjs-vm that are relevant to the VM, to make them modular enough for eWASM to be integrated. This refactoring is occuring hand-in-hand with the modernization effort.

Timeline
  • January 2019
    • ✅ Started refactoring to prepare for future ewasm integration (#424)
    • ✅ Open PR on basic support for ewasm precompiles (#431)
  • February 2019
    • ✅ Refactored memory manipulation of EVM (#442)
    • ✅ Replaced static vm logTable with dynamic inline version in EXP opcode (#450)
  • March 2019
    • ✅ Refactor stack manipulation in EVM (#460)
    • 🛠️ Refactor EVM execution logic, i.e. interpreter (#441)
    • 🛠️ Design and refactor rest of EVM, including message execution (Also see #455)
  • April 2019
    • 🛠️ Rebase EVM changes to the ewasm precompile PR, and merge
    • 🛠️ Experiment with solutions for the sync/async problem
Reason

Project currently stalled due to too large uncertainties on the Ethereum eWASM roadmap. There is some occasional experimentation happening on the sideline though and project might be resumed on a later stage.

Canceled Projects

Move canceled projects here (with some notes on in-between outcome and cancellation reason).

Code of Conduct

Our Pledge

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Our Standards

Examples of behavior that contributes to creating a positive environment include:

  • Using welcoming and inclusive language
  • Being respectful of differing viewpoints and experiences
  • Gracefully accepting constructive criticism
  • Focusing on what is best for the community
  • Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

  • The use of sexualized language or imagery and unwelcome sexual attention or advances
  • Trolling, insulting/derogatory comments, and personal or political attacks
  • Public or private harassment
  • Publishing others’ private information, such as a physical or electronic address, without explicit permission
  • Other conduct which could reasonably be considered inappropriate in a professional setting

Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

Scope

This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.

Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at holger@ethereum.org. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

Indices and tables