Don't get me wrong, I don't like this ESB, I find it messy, confusing and incapable to achieve minimal industrial requirements -- WSO2 Api Manager is a pretty good product, though. This section isn't an apology, merely some tips for those unfortunates who would have or had, like myself, to deal with this product.
When dealing with WSO2 ESB projects, you have to be especially careful with headers, for a lot of sensitive data might leak through them. Let's take an example. Here we have a typical usage of the ESB: expose a REST API to interface some services with a legacy SOAP back-end after some mapping.
| ==========> | | ===========> |
Client | | ESB | | SOAP backend
| <========== | | <=========== |
The backend access is protected with a Basic
authentication. Basically, it requires to add Base64(name_or_the_user:his_password)
to the Authorization
header; like so:
<api context="/constructionSite"
name="crm4sap-constructionSiteApi"
statistics="enable"
trace="enable"
xmlns="http://ws.apache.org/ns/synapse">
<resource faultSequence="crm4sap-errors" methods="POST">
<inSequence>
<sequence key="crm4sap-readEnvVars"/>
<datamapper config="gov:datamapper/map-constructionSite.dmc"
inputSchema="gov:datamapper/map-constructionSite_inputSchema.json"
inputType="JSON"
outputSchema="gov:datamapper/map-constructionSite_outputSchema.json"
outputType="XML"
xsltStyleSheet="gov:datamapper/map-constructionSite_xsltStyleSheet.xml"/>
<log level="full"/>
<header description="Authorization"
expression="fn:concat('Basic ', base64Encode(fn:concat($ctx:sap_user,':',$ctx:sap_password)))"
name="Authorization"
scope="transport"/>
<send>
<endpoint key="gov:endpoints/sap_constructionSite.xml"/>
</send>
</inSequence>
<outSequence>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<respond/>
</outSequence>
</resource>
</api>
Now, let's have a look at the answer
curl -X POST "http://127.0.0.1:8280/constructionSite" --data @env_dev/inputs/input_rsa1.json \
-H "Content-Type: application/json" -v
Note: Unnecessary use of -X or --request, POST is already inferred.
Trying 127.0.0.1...
TCP_NODELAY set
Connected to 127.0.0.1 (127.0.0.1) port 8280 (#0)
POST /constructionSite HTTP/1.1
Host: 127.0.0.1:8280
User-Agent: curl/7.55.1
Accept: */*
Content-Type: application/json
Content-Length: 1306
Expect: 100-continue
HTTP/1.1 100 Continue
We are completely uploaded and fine
HTTP/1.1 200 OK
Authorization: Basic YnRjaF9jcm06c29tZV9wYXNzd29yZA== # DANGER ZONE
activityid: 8370fee1-6189-400e-a28e-2c29bfbd654e
Accept: */*
Expect: 100-continue
Host: 127.0.0.1:8280
Content-Type: application/json; charset=UTF-8
Date: Fri, 05 Jun 2020 08:40:52 GMT
Transfer-Encoding: chunked
{"response":"some response"}* Connection #0 to host 127.0.0.1 left intact
As you can see, headers of the request send to the back-end are implicitly inherited by the out sequence
. This is how we end up with the back-end credentials in the Authorization
This sucks! Unfortunately, you explicitly have to remove those headers before sending anything back to the client (same goes with transport.vfs
variables that can be even more tricky), like so
<api context="/constructionSite"
name="crm4sap-constructionSiteApi"
statistics="enable"
trace="enable"
xmlns="http://ws.apache.org/ns/synapse">
<resource faultSequence="crm4sap-errors" methods="POST">
<inSequence>
<sequence key="crm4sap-readEnvVars"/>
<log level="full"/>
<datamapper config="gov:datamapper/map-constructionSite.dmc"
inputSchema="gov:datamapper/map-constructionSite_inputSchema.json"
inputType="JSON"
outputSchema="gov:datamapper/map-constructionSite_outputSchema.json"
outputType="XML"
xsltStyleSheet="gov:datamapper/map-constructionSite_xsltStyleSheet.xml"/>
<log level="full"/>
<header description="Authorization"
expression="fn:concat('Basic ', base64Encode(fn:concat($ctx:sap_user,':',$ctx:sap_password)))"
name="Authorization"
scope="transport"/>
<send>
<endpoint key="gov:endpoints/sap_constructionSite.xml"/>
</send>
</inSequence>
<outSequence>
<property name="Authorization" scope="transport" action="remove"/> #HERE
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<respond/>
</outSequence>
</resource>
</api>
This way, the back-end's credentials are safe
curl -X POST "http://127.0.0.1:8280/constructionSite" --data @env_dev/inputs/input_rsa1.json \
-H "Content-Type: application/json" -v
Note: Unnecessary use of -X or --request, POST is already inferred.
Trying 127.0.0.1...
TCP_NODELAY set
Connected to 127.0.0.1 (127.0.0.1) port 8280 (#0)
POST /constructionSite HTTP/1.1
Host: 127.0.0.1:8280
User-Agent: curl/7.55.1
Accept: */*
Content-Type: application/json
Content-Length: 1306
Expect: 100-continue
HTTP/1.1 100 Continue
We are completely uploaded and fine
HTTP/1.1 200 OK
activityid: 532851ed-1763-45b2-84fb-1b07b4983512
Accept: */*
Expect: 100-continue
Host: 127.0.0.1:8280
Content-Type: application/json; charset=UTF-8
Date: Fri, 05 Jun 2020 12:51:35 GMT
Transfer-Encoding: chunked
{"response":"some response"}* Connection #0 to host 127.0.0.1 left intact
Here is an other limitation of this product: it does not deal with Oauth authentication so you need to create your own sequence to deal with it. Let's call it auth_oauth
.
Before you call any resource that require oauth authentication, run the following sequence.
<sequence name="auth_oauth" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<enrich description="store">
<source clone="true" type="body"/>
<target property="payload" type="property"/>
</enrich>
<propertyGroup description="REG retrieval">
<property expression="get-property('SYSTEM_TIME')" name="now" scope="default" type="STRING"/>
<property expression="get-property('registry','gov:/endpoints/some_token/valid_until')"
name="valid_until" scope="default" type="STRING"/>
<property expression="get-property('registry','gov:/endpoints/some_token/access_token')"
name="access_token" scope="default" type="STRING"/>
</propertyGroup>
<property action="remove" name="TRANSPORT_HEADERS" scope="axis2"/>
<filter xpath="$ctx:access_token='' or $ctx:now > $ctx:valid_until">
<then>
<log description="TOKEN GEN">
<property name="projectName" value="rsa2wayzz"/>
<property name="action" value="tokenRenewal"/>
</log>
<property name="DISABLE_CHUNKING" scope="axis2" type="STRING" value="true"/>
<header expression="fn:concat('Basic ', base64Encode(fn:concat($ctx:client_id,':',$ctx:client_secret)))"
name="Authorization" scope="transport"/>
<payloadFactory description="i" media-type="text">
<format>grant_type=password&username=$1&password=$2</format>
<args>
<arg evaluator="xml" expression="$ctx:user"/>
<arg evaluator="xml" expression="$ctx:password"/>
</args>
</payloadFactory>
<call blocking="true">
<endpoint>
<http method="post" uri-template="https://some/api/token">
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>-1</progressionFactor>
<maximumDuration>0</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</http>
</endpoint>
</call>
<property expression="json-eval($.expires_in)" name="expires_in" scope="default" type="STRING"/>
<script language="js">
<![CDATA[var sum = (parseInt(mc.getProperty("expires_in")) - 3) * 1000 + parseInt(mc.getProperty("now")) ;
var sum = mc.setProperty("valid_until",sum.toString());]]>
</script>
<propertyGroup description="set token">
<property expression="json-eval($.access_token)"
name="gov:/endpoints/some_token/access_token" scope="registry" type="STRING"/>
<property expression="$ctx:valid_until"
name="gov:/endpoints/some_token/valid_until" scope="registry" type="STRING"/>
</propertyGroup>
</then>
<else/>
</filter>
<enrich description="restore">
<source clone="true" property="payload" type="property"/>
<target type="body"/>
</enrich>
</sequence>
The general idea is the following. First, store the payload with enrich
, to avoid it to be send to the token renewal url instead of the endpoint you'll call later on. Then, look for the reg:.../valid_until
value that this very same sequence might have written earlier. Compare it with SYSTEM_TIME
(a.k.a. $ctx:now
) during the filter
phase. Either the token is still valid, and you are good to go or you need to renew the token.
Depending on the API, this phase might change from one API to another. In this example, the API needs both base64(client_id:client_secret)
in the Authorization: Basic
header and some user and password to issue a token. The successful response looks like this:
{
"access_token": "some_token",
"valid_until": 9000
}
The access_token
will be the one used while calling for the resource later on.
<header description="Set Authorization" expression="fn:concat('Bearer ',$ctx:access_token)" name="Authorization" scope="transport"/>
But to be really useful, we'll need to store this token along with the precise moment when it will be outdated. We need to compute this based on SYSTEM_TIME
(which is in milliseconds: thus * 1000
). This is where the script
mediator is required. Last but not least, store both valid_until
and access_token
in registry variables
Facts:
Here is the most straightforward way I found to achieve these goals with Gitlab
Besides versioning your ESB projects (in this case, with Gitlab), you will need to take care of the following details
The following example requires that the different modules of your projects
ESBConfigProject
CompositApplicationProject
ConnectorExporterProject
RegistryRessourceProject
are explicitly linked in a mavenmultimodule
pom.xml
. Here is an example of such a pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>com.someCompany</groupId>
<artifactId>someProject</artifactId>
<version>1.0.0</version>
<packaging>pom</packaging>
<name>someProject</name>
<description>some description</description>
<modules>
<module>someProject_ESB</module>
<module>someProject_REG</module>
<module>someProject_CE</module>
<module>someProject_CA</module>
</modules>
<build>
<plugins>
<plugin>
<artifactId>maven-eclipse-plugin</artifactId>
<version>2.9</version>
<configuration>
<buildcommands />
<projectnatures>
<projectnature>org.wso2.developerstudio.eclipse.mavenmultimodule.project.nature</projectnature>
</projectnatures>
</configuration>
</plugin>
</plugins>
</build>
</project>
Except for the CompositApplicationProject
name of which should end with _CA
, you are free to use different names from one project to another provided that all your project share the following folder structure:
[someProject]
pom.xml (the one above)
.gitlab-ci.yml (see bellow)
[someProject_CA]
pom.xml
.....
[someProject_ESB]
pom.xml
src
main
....
.....
[someProject_CE]
pom.xml
.....
[someProject_REG]
pom.xml
.....
You'll also need on premise gitlab runners (shell type) with write rights on you $WSO2_EI_HOME/repository/deployment/server/carbonapps
. That's all you need. installation details
.gitlab-ci.yml
build_car:
stage: build
image: maven:3.6.3-jdk-8
only:
- master
- dev
script:
- export CI_PROJECT_NAME
- echo "Starting build ${CI_PROJECT_NAME}"
- mvn clean install -Dmaven.test.skip=true -Dversion=$CI_COMMIT_SHORT_SHA
artifacts:
paths:
- ./*_CA/target/*.car
tags:
- docker
dev_deploy:
stage: deploy
script:
- $carbon_app_name = $(Get-ChildItem ./*_CA/target/*.car).name.split('_')[0]
- $old = Get-ChildItem $carbon_app_name*.car -Path "$($env:WSO2_EI_HOME)\repository\deployment\server\carbonapps"
- if ($old) {rm $old.fullname}
- $new = Get-ChildItem ./*_CA/target/*
- Copy-Item -Path $new -Destination "$($env:WSO2_EI_HOME)\repository\deployment\server\carbonapps"
only:
- dev
tags:
- wso2
- esb
- dev
production_deploy:
stage: deploy
script:
- $carbon_app_name = $(Get-ChildItem ./*_CA/target/*.car).name.split('_')[0]
- $old = Get-ChildItem $carbon_app_name*.car -Path "$($env:WSO2_EI_HOME)\repository\deployment\server\carbonapps"
- if ($old) {rm $old.fullname}
- $new = Get-ChildItem ./*_CA/target/*
- Copy-Item -Path $new -Destination "$($env:WSO2_EI_HOME)\repository\deployment\server\carbonapps"
only:
- master
tags:
- wso2
- esb
- prod
The following pipeline has two steps.
The build_car
job will be realized by shared runners for every new commit on either the dev
or master
branches. Here, we are using maven:3.6.3-jdk-8
official docker image. Here you probably noticed that we are using -Dversion=$CI_COMMIT_SHORT_SHA
during mvn clean install
. That's because we are using variables in pom.xml
's versions, like so:
<project
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.someCompany</groupId>
<artifactId>someProject</artifactId>
<version>1.0.0</version>
</parent>
<groupId>com.someCompany.someProject_CA</groupId>
<artifactId>someProject_CA</artifactId>
<version>${version}</version>
<packaging>carbon/application</packaging>
<name>someProject_CA</name>
<description>someProject_CA</description>
(you'll understand later why). Once build, the .car
is stored in the project's artifacts
Depending on whether trigged by the dev
or prod
branch, the success of the build
job will trigger the deploy
job for the relevant gitlab-runner. In this powershell based example, the runner
.car
's name (e.g. someProject_CA_c5ad90bf.car
-> someProject
)$WSO2_EI_HOME/repository/deployment/server/carbonapps