ejbconcurrency

ejbconcurrency is a small project for checking how container handles EJB transactions in @Stateless and @Singleton session beans. We see the difference between using @EntityManager and @Datasource for database interactions.

Project structure

The project consists in three modules:

• ejbconcurrency-server EJB to be deployed in the application server.
• ejbconcurrency-api API exposed by the EJB (Remote and DTOs used by the client are located here).
• ejb-client standalone java application for testing the EJB.

EJB - Description

Dashboard is a stateless EJB which adds a point in the dashboard. If any other free point present, adds a new segment between two points. The same point can not be used in two segments.

EJB - Technical

@Stateless DashboardRemote.addPointToDashboard("A1") => Adds the segment [A0-A1], with the first free point A0 found. Given a new incoming point A1:

• There is any free point A0 in database? => select * from POINT where rownum=1;
  • YES
    1. Creates the segment A0-A1 => insert into table SEGMENT(A0, A1)
    2. Remove A0 from the temporary table => delete from POINT where name = 'A0'
  • NO
    1. Add A1 to the temporary table => insert into table POINT values(A1)
    2. A1 will be coupled with the next eventually coming point, A2.

If a client calls DashboardRemote and sends a sequence of points, one by one: A0, A1, A2, A3, A4, A5, A6.
At the end of the computation the data in DB will be as follow:

Segments: [A0-A1], [A2-A3], [A4-A5]
Points:   [A6] 

Concurrent scenario

If two clients calling concurrently (at the same time) DashboardRemote with the following sequences:

CLIENT1: A1, A2, A3, A4, A5
CLIENT2: B1, B2, B3, B4, B5

At the end of the operation, one possible scenario could be:

Segments: [A1-B1], [A2-A3], [B2-A4], [B3-A5], [B4-B5]

Calls from CL1 and CL2 will be processed concurrently so various coupling(combinations) can hapen.
To each client the container will assign a different instance of the STLB from the pool, lets say INSTACE1 will compute request from CLIENT1 and INSTANCE2 will compute request from CLIENT2.
IMPORTANT: Same point should not be in two different segments.

Problem

Consider A0 is present in POINT db table and two new requests are coming, point A1 from CLIENT1 and point B1 from CLIENT2. Steps to be done for completing the task on each SLSB instance are the following:

1. select * from POINT where rownum=1
2. insert into table SEGMENT(A0, A1)
3. delete from POINT where name = 'A0'

We want:

• Three steps to be in a single transaction, so failure of a single instruction/step will rollback all steps.
• Same point can not be present in two different segments.
• We don't want any point to remain uncoupled if another one is free.

We want to avoid this scenario:

1. CL1 executes step1, he find A0.
2. CL2 executes step1, he find A0 <= both found A0 as free.
3. CL1 executes step2 <= [A0-A1] will be inserted in DB. OK
4. CL2 executes step2 <= [A0-B1] will be inserted in DB. NOTOK, A0 is not anymore free.

Lets see how the container will handle this situation in different scenarios. See also Isolation Levels

Test configuration

SERVER
Between step1 and step2 we add Thread.sleep(1000) in the EJB. We assume the computational time from step1 to step2 is 1 second long.

CLIENT
It creates two Threads, one for client CLI1 and one for CLE2, which send "in parallel" the following sequences to the server:

CL1: A1, A2, A3, A4, A5, A6, A7, A8, A9, A10
CL2: B1, B2, B3, B4, B5, B6, B7, B8, B9, B10

Use interface DashboardRemote.java for calling the EJB, specifying one of the following implementations.

Different approaches	EJB Implementation
@PersistenceContext EntityManager	DashboardEM.java
@Datasource	DashboardDS.java
Datasource inside @Singleton	DashboardDSUseSingleton.java
EntityManager in ejb-service-dao approach	DashboardEMSafe.java

Test 1 - Using EntityManager

@PersistenceContext EntityManager entityManager

RESULT

SEGMENTS with no re-send: [A0-B1], [B0-B2], [B3-B4], [B5-A4], [A5-B7], [B8-A7], [A8-A9]
SEGMENTS with re-send:    [A0-B1], [B0-B2], [B3-B4], [B5-B6], [B7-B8], [B9-A1], [A2-A3], [A4-A5], [A6-A7], [A8-A9]
Points:[]   
Missing points with no re-send: A1, A2, A3, A6, B6, B9
Missing points with re-send: 0

OBSERVATION

• When transaction(INSTANCE2) tries to changes data changed by another transaction(INSTANCE1) in meanwhile, the container throws the exception:
  javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
  - Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
  - ...
  - Caused by: javax.persistence.OptimisticLockException: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1
  - Caused by: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1
• For consuming all the points (even the one which failed) in the client we added a re-sending mechanism like:
  while (!success && count < 10) {remote.addPointToDashboard("name");}. If the bean is a MDBthere is no need for this mechanism, as MBD (Messaging broker actually) comes with a built-in resending policy, and on EJBTransactionRolledbackException the container will resend the message again.

Test2 - Using Datasource

@Resource(mappedName = "java:jboss/datasources/ExampleDS") private DataSource dataSource;

RESULT

SEGMENTS:[A0-A1], [A0-B1], [B0-A2], [B0-B2], [B3-A4], [B3-B4], [A3-A5], [A3-B5], [A6-B6], [A7-B7], [A8-B8], [A9-B9]
Points: []

OBSERVATION

• The result is not correct, duplicated points!
• @Datasource does not offer any isolation level control, so threads/instances are accessing concurrently to the DB tables.
• SOLUTION1: Use DB layer isolation level to avoid Nonrepeatable and Phantoms Reads. select for update in oracle.
• SOLUTION2: Control the isolation level of the connection. Same connection has to be used for all 3 steps.
  Connection conn = dataSource.getConnection();
conn.setAutoCommit(false);
conn.setTransactionIsolation(Connection.TRANSACTION_READ_COMMITTED);
step1(conn)
step2(conn)
step3(conn)
conn.commit();
• SOLUTION3*: Use of Java mechanism to implement SERIALIZABLE isolation level so only one thread at time can execute all transaction steps. Notice that to do so in a SLSB we need a synchronized block/method and a static variable to be shared between all instances (to be used like a lock). This approach could be the worse one, since usage of static variables and saving the state in SLSB is discouraged by EJB spec. Also, locking access never improves on access times. The cleanest way to do it, starting from EJB3.1 is using SL @Singleton. (see TEST3)

TEST3 - Using @Singleton

Using singleton with WRITE lock we make the isolation level to SERIALIZE.

RESULT

SEGMENTS:  [A0-B0], [A1-A2], [B1-A3], [B2-A4], [B3-A5], [B4-A6], [B5-A7], [B6-A8], [B7-A9], [B8-B9]
Points: []
Missing points: []

Observations

• Using the highest level of isolation in the method addPointToDashboard we make possible only one thread at a time can access addPointToDashboard().
• Creates a bottle neck, all the threads (clients) have to wait after each other in order to execute the method.
• Execution time increases to 20 seconds, sending 10 points, versus 10 seconds in case of EntityManager. Try running:
  $ java -jar dashboardclient.jar 1 10
$ java -jar dashboardclient.jar 3 10
  
• Lose the benefit of having different stateless instances in the pool.

Environment

• Application Server: Jboss7.1.1 using standalone-full.xml profile. No additional configuration download page version.
• Datasource: ExampleDS present in standalone-full.xml which connects to in memory H2 database.

Deployment

In the dist folder you can find already builded modules for running the test.

• ejbconcurrency-ear.ear - to be deployed on jboss
• dashboardclient.jar - to test the EJB from the command line

Example, to call the EJB using the implementation with EntityManager and send 10 points to the dashboard use the following command:

$ java -jar dashboardclient.jar 1 10

INFO [Main] .main(32) | Calling EJB Segment with implementation "DashboardEM". Created 2 clients, each of them will send to the EJB 10 points
INFO [Main] .main(33) | We expect at the end of the test: 10 Segments and 0 Points
INFO [DashboardClient] .resetDashboard(111) | CLIENT-CLEANER: Dashboard cleaned
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A0
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B0
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A1
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point B1): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A2
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A3
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point B1): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A4
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A5
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point B1): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B1
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B2
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point A6): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B3
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B4
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point A6): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A6
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A7
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point B5): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B5
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B6
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point A8): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B7
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B8
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point A8): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-B - Added point: B9
ERROR[DashboardClient] .addPointToDashboard(80) | General Exception (retry for point A8): Transaction rolled back - Cause: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A8
INFO [DashboardClient] .addPointToDashboard(77) | CLIENT-A - Added point: A9
INFO [Main] .main(56) | Completed in 10946 ms, 10 secs
INFO [DashboardClient] .getAllSegments(90) | CLIENT-clientCheck - Segments: [[A0-A1], [B0-A2], [A3-A4], [A5-B1], [B2-B3], [B4-A6], [A7-B5], [B6-B7], [B8-B9], [A8-A9]]
INFO [DashboardClient] .getAllPoints(101) | CLIENT-clientCheck - points: []
INFO [Main] .checkData(68) | nrClients: 2, nrCallsForClient: 10, nr segments created: 10, nr single points: 0
INFO [Main] .checkData(75) | Segments OK. No duplicates found.
INFO [Main] .checkData(81) | Points OK. All point are coupled.

Conclusions

• SLSB is safe in sense that container guarantees only one thread at time can execute a single instance. SPEC do not mention that the same method will be called only from 1 instance at time. (controversal answers about this)
• EntityManager is reliable. With the default isolation level is enough to avoid data inconsistent.
• Nice to see how to change isolation level and find an extreme case when needed. Even if from SPEC 13.3.2 "Isolation Levels Therefore, the EJB architecture does not define an API for managing isolation levels."
• A "copy" of EnityManager is injected to each SLBS instances by the container. Afterwards it is the EntityManager responsible for data consistency.

See

• JSR-318 EJB 3.1 Spec - Enterprise JavaBean 3.1 Specification (Cap13. Support for Transactions, 13.3.2 Isolation Levels)
• Adam Bien's weblog - Is EntityManager Thread Safe?
• Concurrency in a Singleton - 34.2.1.2 Managing Concurrent Access in a Singleton Session Bean
• Isolation Levels - Real example of Dirty Read, Phantom Read and Non Repeatable Read
• Reentrant Lock - Simple explanation of reentrant locks in Java.
• OptimisticLockException - Example when OptimisticLockException is thrown.
• @Stateless or @Singleton instead of static helper class? Stackoverflow post. Why this project.
• Why Stateless session beans? - Stackoverflow post. See BalusC and user698226 answers.
• Is EntityManager really thread-safe? - Stackoverflow post. See the accepted answer.