|
| 1 | +--- |
| 2 | +id: apachebeam |
| 3 | +title: Apache Beam Runner |
| 4 | +sidebar_label: Apache Beam |
| 5 | +--- |
| 6 | + |
| 7 | + |
| 8 | +Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. |
| 9 | +Beam provides several open sources SDK's which can he used to build a pipline. This pipline is then |
| 10 | +executed using one of the many distributed processing back-ends supported by Apache beam. Currently |
| 11 | +Frameworks such as Apache Spark and Apache Flink are supported, these are called runners. |
| 12 | + |
| 13 | +Twister2 has a Apache Beam runner that is currently being shipped with the Twister2 release, which we plan to |
| 14 | +include directly into Apache Beam, This runner allows users to run Apache beam piplines with Twister2 |
| 15 | + |
| 16 | +Below is an example code that runs a Beam pipline using Twister2 Runner |
| 17 | + |
| 18 | +```java |
| 19 | + Twister2PipelineOptions options = PipelineOptionsFactory.as(Twister2PipelineOptions.class); |
| 20 | + options.setTSetEnvironment(env); |
| 21 | + options.as(Twister2PipelineOptions.class).setRunner(Twister2LegacyRunner.class); |
| 22 | + String resultPath = "/home/pulasthi/work/twister2/beamtest/testdir"; |
| 23 | + Pipeline p = Pipeline.create(options); |
| 24 | + PCollection<String> result = |
| 25 | + p.apply(GenerateSequence.from(0).to(10)) |
| 26 | + .apply( |
| 27 | + ParDo.of( |
| 28 | + new DoFn<Long, String>() { |
| 29 | + @ProcessElement |
| 30 | + public void processElement(ProcessContext c) throws Exception { |
| 31 | + c.output(c.element().toString()); |
| 32 | + } |
| 33 | + })); |
| 34 | + |
| 35 | + try { |
| 36 | + result.apply(TextIO.write().to(new URI(resultPath).getPath() + "/part")); |
| 37 | + } catch (URISyntaxException e) { |
| 38 | + LOG.info(e.getMessage()); |
| 39 | + } |
| 40 | + p.run(); |
| 41 | +``` |
| 42 | + |
| 43 | +Apache beam provides a rich set of SDK's with many adapters which allow you to integrate with |
| 44 | +various systems, you can learn more about Apache Beam by referring the Apache Beam [documentation](https://beam.apache.org/documentation/). |
0 commit comments