Skip to content

Commit 3fa7c50

Browse files
committed
adding beam doc
1 parent 2faf4d9 commit 3fa7c50

File tree

1 file changed

+44
-0
lines changed

1 file changed

+44
-0
lines changed

docs/docs/compatibility/beam.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
id: apachebeam
3+
title: Apache Beam Runner
4+
sidebar_label: Apache Beam
5+
---
6+
7+
8+
Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines.
9+
Beam provides several open sources SDK's which can he used to build a pipline. This pipline is then
10+
executed using one of the many distributed processing back-ends supported by Apache beam. Currently
11+
Frameworks such as Apache Spark and Apache Flink are supported, these are called runners.
12+
13+
Twister2 has a Apache Beam runner that is currently being shipped with the Twister2 release, which we plan to
14+
include directly into Apache Beam, This runner allows users to run Apache beam piplines with Twister2
15+
16+
Below is an example code that runs a Beam pipline using Twister2 Runner
17+
18+
```java
19+
Twister2PipelineOptions options = PipelineOptionsFactory.as(Twister2PipelineOptions.class);
20+
options.setTSetEnvironment(env);
21+
options.as(Twister2PipelineOptions.class).setRunner(Twister2LegacyRunner.class);
22+
String resultPath = "/home/pulasthi/work/twister2/beamtest/testdir";
23+
Pipeline p = Pipeline.create(options);
24+
PCollection<String> result =
25+
p.apply(GenerateSequence.from(0).to(10))
26+
.apply(
27+
ParDo.of(
28+
new DoFn<Long, String>() {
29+
@ProcessElement
30+
public void processElement(ProcessContext c) throws Exception {
31+
c.output(c.element().toString());
32+
}
33+
}));
34+
35+
try {
36+
result.apply(TextIO.write().to(new URI(resultPath).getPath() + "/part"));
37+
} catch (URISyntaxException e) {
38+
LOG.info(e.getMessage());
39+
}
40+
p.run();
41+
```
42+
43+
Apache beam provides a rich set of SDK's with many adapters which allow you to integrate with
44+
various systems, you can learn more about Apache Beam by referring the Apache Beam [documentation](https://beam.apache.org/documentation/).

0 commit comments

Comments
 (0)