# How to join the Sandbox Dataspace

## 0. Prerequisites

### Server / network infrastructure

Your local connector will be accessible under a https-URL of your company. Therefore, you need:

- a subdomain and/or path as the address for the connector
- a proxy infrastructure that can handle TLS certificates and forward the requests for that subdomain and/or path to an internal address   

In this description, as an example, we assume that the new connector will be accessible under https://connector.my-company.com. 

### Server infrastructure

- You will need a (virtual) server running Linux (e.g. Ubuntu), with installed Docker engine v28.3.x or higher.
- To access it, you need a tool for ssh connections on your client (PuTTY or similar). 

### Tools for testing (on the client)

- A tool to run GET/POST requests. In this repository, we have two versions:
  - There are `.http` files which can be executed with the http plugin of IntelliJ. 
    There is also a plugin for VisualCode which can use these `.http` files.  
  - Otherwise, you can use Postman and the `.postman_collection.json` files. 

- If you use Windows to change configuration files, store them with Unix file endings (only \n, not \r\n as usual in Windows)!
  To achieve this, a tool like Notepad++ or similar is useful. Windows line endings can be harmless in some cases (e.g. 
  for docker-compose.yml), but in other files they might cause problems that are very hard to track down! (Messages that
  mention the character "\r" like `unexpected string "whatever\r"` are a sign of a problem caused by Windows line endings.)
  Alternatively, you edit the configuration files directly on the server (using e.g. "nano" or "vim"). 

- (Optional) To check or adjust database content, the database plugin of IntelliJ could be convenient, but any other 
  tool that can access Postgres databases is ok. Here, we use the command line tool of Postgres directly in the Docker 
  containers, so no additional tool is necessary. 
 

### URL / DID / Participant ID

The DID ("Decentralized Identifier") is the unique identifier of your connector.  
A DID can be converted into a URL, and under this URL, the DID (containing all necessary information for that 
connector) must be publicly available. Therefore, you cannot freely "invent" your connector's DID. Instead, it is 
defined by the URL where your connector will run. 
This depends on your company's guidelines and infrastructure:

- You might use a subdomain of your company URL: e.g. https://connector.my-company.com (we recommend this approach)
- or a path for an existing domain or subdomain: e.g. https://www.my-company.com/conn

- Note your connector's future URL here: https://____________________________________________

The DID will be accessible under this URL at path `/did`.  
To find the DID, convert your connector's URL as follows:

1. replace the starting `https://` by `did:web:`
2. if you have a path, replace all slashes (`/`) in the path by colons (`:`)
3. append `:did`

In this description, we assume the example URL https://connector.my-company.com - this becomes `did:web:connector.my-company.com:did`.

- Note your connector's DID here: did:web:____________________________________________:did

You also need to define a `participant_id`: It should be a short but unique ID for the connector. You can include e.g. 
your company name, like `myCompany-connector`. 

- Note your connector's participant_id here: ___________________________________


## 1. Server setup: networking

Goal of this step is to have an "empty" nginx container that is accessible under the URL of the later connector.
(This nginx container will later be replaced by the real one doing the routing for all connector components.)

You need:

- The URL (subdomain/domain/path) for the connector (as defined in section [URL / DID / Participant ID](#url--did--participant-id)).
- A (virtual) server where you can install Docker.

## 2. Server setup: connector

### 2.1 Import Docker image

Use the connector image provided with this setup:  `wilson-embedded-connector-0.5.1.tar`

Alternatively, if you have done software changes to the connector, you can use your local docker image (save it with 
`docker image save -o myconnector.tar <name of your built image e.g. wilson-embedded-connector:0.5.1>`). 
Not recommended for the first connector start! 

1. Copy the `.tar` file to the server

2. On the server, import it the image to docker using the following command: 
   `(sudo) docker load --input wilson-embedded-connector-0.5.1.tar`
   
### 2.2 Adapt setup files

These steps are done on your local machine, not on the server.

#### Automatic generation of setup files (recommended)

There are two versions of this script (the do the same):

- For Windows, open a prompt in the same directory as this `README.md` file, then start the file generation with the 
  command `powershell -ExecutionPolicy Bypass -File .\1_createConnectorFiles.ps1`.
- For Linux, open a bash shell in the same directory as this `README.md` file, then start the file generation with the
  command `./1_createConnectorFiles.sh`.

Both will generate a directory `_new-connector` containing files for your new connector. The subdirectory `forServer` 
is the one that you will copy to the server later. Proceed as described in [section 2.3](#23-setup-of-the-connector-components), automatic method.

### 2.3 Setup of the connector components

This section describes how to start the containers and finish their configuration. 
You will need an ssh connection to your server.

#### Automatic setup of connector components (recommended)

- Copy the content of the subdirectory `forServer` to your server (into subdirectory `connector`), e.g. using scp or FileZilla.
- Open an ssh connection to the server and change into subdirectory `connector` (this is the directory of `docker-compose.yml`).
- Make the scripts executable: `chmod +x *.sh`

Then: 

1. Run the script `./2_initVault+Database.sh`. This will initialize and unseal the vault, will init the database and 
   start the connector. Read the script output if everything is ok.  

2. Check if the connector container is `healthy`:
    - `sudo docker container ls`, you should see the connector container (the one with image `wilson-embedded-connector:0.5.1`)
      with a status `Up ... (healthy)`.
    - If not healthy, wait a minute. If still not healthy, check the container log, e.g. for messages about wrong
      database password (if so, check or re-do the database setup). See the troubleshooting tips at the end of this 
      document for information on how to check the logs etc. . 
    - Some error messages like `java.lang.IllegalArgumentException: Key 'renewable' does not exist` are normal during
      the first start. But near the end, it should display a message like `INFO: Runtime ...some-id... ready`.

3. If the connector is healthy: 
    - Copy the values from file `vaultSecrets.txt` from the server to safe place. Important are the "unseal keys" 
      (5 values) and the "Initial Root Token" (1 value). After that, you can delete the file on the server.
    - Delete the `sql` directory on the server (`sudo rm sql/*, sudo rmdir sql`).

   If the connector is not healthy or other problems occur, use the script `0_down-all.sh` to stop all containers, then 
   fix the problems. If you have to adapt files, adapt them on your local computer!   
   Before running again the script `2_initVault+Database.sh`, you MUST delete the content of the directory `connector` 
   on the server and copy the files `forServer` again from your computer to the server! (This is important because the 
   script replaces placeholders in `.env` and other files with new values like the generated vault key. This won't be 
   done if the placeholders have already been replaced, leaving you with a mix of old and new keys.) 

4. Only if the connector is healthy: run `./3_initConnector.sh`.
   - On success (no error messages in script, vault container running, db and connector containers healthy), 
     continue with the steps in section [3.4](#34-configure-participants-identity-etc)
   - If nginx cannot be started and the message indicates "port already in use" or similar, check the port number in the
     `nginx` part of the `docker-compose.yml` file. Refer to section [1.2 Start temporary nginx](#12-start-temporary-nginx) 
     for useful hints. After adapting the port number, restart nginx with `sudo docker compose up nginx`.  

Continue as described in section [2.4 Configure participants, identity etc.](#24-configure-participants-identity-etc).

### 2.4 Configure participants, identity etc.

Now you need to run some POST (and GET) requests:

- If you can use the .http-plugin of IntelliJ or VisualCode, use the calls defined in the `.http` file 
  [http/4_connector-setup.http](_new-connector/http/4_connector-setup.http). Before running, you need to select the  
  environment: in the IDE, above the file, is must show an environment, not "no environment".
- Otherwise, use postman with the calls defined in the postman JSON file 
  [postman/4_connector-setup.postman_collection.json](_new-connector/postman/4_connector-setup.postman_collection.json).

**Important:** Make sure you run the file from the `_new-connector` directory, not from the `connector-template`! 

Both files also contain some test calls.    
Run the calls one by one, be sure to read the comments and do as stated there (e.g. one step creates output which you should store somewhere). 

If there are no errors, you are is done - the connector is up and running!  


### 2.5 First test of the connector

1. To test the connector as data consumer, you can test a first contract negotiation and data transfer with the steps 
   in [http/5_test-consume-data.http](_new-connector/http/5_test-consume-data.http) or
   [postman/5_test-consume-data.postman_collection.json](_new-connector/postman/5_test-consume-data.postman_collection.json)  

2. To test the connector as data provider, you can test publishing a first offer with the steps in 
   [http/6_test-provide-data.http](_new-connector/http/6_test-provide-data.http) or
   [postman/6_test-provide-data.postman_collection.json](_new-connector/postman/6_test-provide-data.postman_collection.json)
   - You can use the provided demo URL, or use a different one (must be a URL that does not require authentication).
   - Refer to the description of the calls for details. 
