# Development
# Contributing
All code changes should be done through issues and merge requests in https://gitlab.com/alleles/ella.
Issues should be discussed with core developers/project owner before being started.
# Getting started
The recommended way of developing with ELLA is to use VSCode with development containers. For this to work you need to follow the recipe provided here (opens new window).
Once VSCode is set up, open the root folder (ella) in VSCode, and use the command palette to launch Remote-Containers: Rebuild and Reopen in Container. From here you will have the choice of three different development containers. ELLA backend for working with the backend code in Python (or legacy frontend), ELLA frontend for working with the frontend code, and ELLA BDD tests for working with the behaviour driven development tests. All development containers mount in the source code under /ella - the differences are what extensions are available. All development containers can run in parallel, by using the option Remote-Containers: Reopen in Container.
Now, ELLA should be available on http://localhost:8080. Log in with credentials testuser1:demo, testuser2:demo, ..., testuser8:demo (see Users). An alternative, as of yet incomplete, frontend is available at http://react.localhost:8080.
To get visibility into what's happening in the browser client, start the Cerebral debugger (opens new window). Enter any name ('ella' is a good name) and port 8585. This sets up a server listening on that part port. Open the app in the browser (refresh if the app was opened before starting Cerebral). The browser will connect to the Cerebral debugger. To enable the web client to connect to the cerebral debugger, you need to comment out this line in ./frontend-legacy/src/js/index.js:
// comment out this line
// config.devtools = null
Also make sure the server port match the port configured in frontend-legacy/src/js/index.js.
# Code style and formatting
We rely on opinionated formatters and linters to ensure consistency in coding style. We rely on pre-commit (opens new window), which runs in the CI to ensure quality is consistent. It is strongly adviced to install this in the repo by following the instructions from https://pre-commit.com/.
See .pre-commit-config.yaml for enforced formatting and linting rules.
# Ops
Bringing up the production/development/demo/testing systems are handled by docker-compose - see YAML files for detailed information.
# Testdata
Testdata in the dev environment is handled by a separate API. On first startup it populates the database with the default testset, but other options are available. See backend/testdata-api/README.md for details.
# Tests
A variety of tests are run as part of the CI. These can also be run manually. See .gitlab-ci.yml for detailed commands to run.
All new features should be covered by unit tests. For bigger features affecting end users, e2e tests should also be included.
Test frameworks used are:
- pytest for unit and integration tests of the backend
- jest for unit tests of the frontend
- WebdriverIO/Selenium for e2e tests of legacy frontend
- behaving (on top of WebdriverIO/Selenium) for behaviour driven development tests of new frontend
# Migration
Whenever you make changes to the database model, you need to create migration scripts, so that the production database can be upgraded to the new version. We use Alembic to assist creating those scripts. Migration scripts are stored in src/vardb/datamodel/migration/alembic/. The current migration base is stored in src/vardb/datamodel/migration/ci_migration_base. This base serves as the base for which the migration scripts will be built against, and should represent the oldest database in production.
# Create a new migration
- Make all your changes to the normal datamodel in
src/vardb/datamodel/and test them until you're satisfied. In general we don't want to make more migration scripts than necessary, so make sure things are proper. - Make and enter a dev instance (not necessary if inside a devcontainer).
- Inside it do:
ella-cli dev reset-database --testset empty --migration(resets database to the migration base, then runs all the migrations)cd /ella/backend/src/vardb/datamodel/migration/PYTHONPATH=../../.. alembic revision --autogenerate -m "Name of migration". This will look at the current datamodel and compare it against the database state, generating a migration script from the differences.
- Go over the created script, clean it up and test it (
MIGRATION=1 pytest).
The migration scripts are far from perfect, so you need some knowledge of SQLAlchemy and Postgres to get it right. Known issues are Sequences and ENUMs, which have to be taken care of manually. Also remember to convert any data present in the database if necessary.
The test-api-migration part of the CI will also test database migrations, by running the api tests on a migrated database.
# Manually testing the migrations
To manually test the migration scripts you can run the upgrade/downgrade parts of the various migrations:
ella-cli database ci-migration-base
ella-cli database upgrade e371dfeb38c1
ella-cli database upgrade 94a80b8848df
ella-cli database downgrade e371dfeb38c1
For migrations involving user generated data, it would be useful to run the migrations (both upgrade and downgrade) with the database populated through "real" use.
NOTE
For these commands to work, the database needs to be reset with migration enabled. See Testdata.
# Integrating the new migration step
Once you are satisfied that the newly created migration script does its job, you can add it to the repository.
# API documentation
The API is documented using apidocs (opens new window) supporting the OpenAPI specification (f.k.a. Swagger 2.0). You can see the specification here (opens new window).
You can explore the ELLA's API at /api/v1/docs/ in you browser. Under the hood, the resources and definitions (models) are loaded into apispec in api/v1/docs.py. The spec is made available at /api/v1/specs/. The definitions are generated automatically by apispec using it's Marshmallow plugin.
To document your resource, have a look at the current resources to see usage examples.
WARNING
The Swagger docs are known to be outdated and not to be trusted.