Come over to our first Elixir meet-up! We're meeting in person in the Software Mansion office. We prepared three Elixir-related presentations, after which you can relax with a bite of pizza and a bottle of beer.
You can also join our live stream here: https://www.crowdcast.io/e/elixir-community-krakow-1 in case you can't come in person.

small_676x380_1_b7e33f7b00.webp

thumbnail_676x380_1_b7e33f7b00.webp

676x380_1_b7e33f7b00.webp

Elixir Community Kraków #1

We're happy to invite you to the second Elixir Community Kraków meet-up! Like the last time, we'll meet in the Software Mansion's office, but we'll be live streaming as well on Youtube: https://www.youtube.com/channel/UCKgDsBDpYdZf2mbRzcEC8FA

small_676x380_f219ed72a2.webp

thumbnail_676x380_f219ed72a2.webp

676x380_f219ed72a2.webp

Elixir Community Kraków #2

This tutorial explains how membrane works. It covers environment preparation, pipelines, and the basic concepts behind the framework.

Hello there, and a warm welcome to the Membrane tutorials. We're glad you chose to learn Membrane; and we'd like to invite you on a journey around multimedia with us, where we explore how to utilize the Membrane Framework to build applications that process audio, video, and other multimedia content in interesting ways.

## What is Membrane?

Membrane is a multimedia processing framework that focuses on reliability, concurrency, and scalability. It is primarily written in Elixir, while some platform-specific or time-constrained parts are written in Rust C. With a range of existing packages and an easy-to-use interface for writing your own, Membrane can be used to process almost any type of multimedia, for example:
- stream via WebRTC, RTSP, RTMP, HLS, HTTP and other protocols,
- transcode, mix and apply custom processing of video & audio,
- accept and generate / record to MP4, MKV, FLV and other containers,
- handle dynamically connecting and disconnecting streams,
- seamlessly scale and recover from errors,
- do whatever you imagine if you implement it yourself :D Membrane makes it easy to plug in your code at almost any point of processing.

If the abbreviations above don't ring any bells, don't worry - this tutorial doesn't require any multimedia-specific knowledge. Then, [other tutorials](https://membrane.stream/learn) and [demos](https://github.com/membraneframework/membrane_demo) will introduce you to the multimedia world!

## Structure of the framework

The most basic media processing entities of Membrane are `Element`s. An element might be able, for example, to mux incoming audio and video streams into MP4, or play raw audio using your sound card. You can create elements yourself, or choose from the ones provided by the framework.

Elements can be organized into a pipeline - a sequence of linked elements that perform a specific task. For example, a pipeline might receive an incoming RTSP stream from a webcam and convert it to an HLS stream, or act as a selective forwarding unit (SFU) to implement your own videoconferencing room. You'll see how to create a pipeline in the subsequent chapter.

### Membrane packages

To embrace modularity, Membrane is delivered to you in multiple packages, including plugins, formats, core and standalone libraries. The list of all Membrane packages is available [here](https://github.com/membraneframework/membrane_core/#All-packages). It contains all the packages maintained by the Membrane team and some third-party packages.

**Plugins**

Plugins provide elements that you can use in your pipeline. Each plugin lives in a `membrane_X_plugin` repository, where `X` can be a protocol, codec, container or functionality, for example [mebrane_opus_plugin](https://github.com/membraneframework/membrane_opus_plugin). Plugins wrapping a tool or library are named `membrane_X_LIBRARYNAME_plugin` or just `membrane_LIBRARYNAME_plugin`, like [membrane_mp3_mad_plugin](https://github.com/membraneframework/membrane_mp3_mad_plugin). Plugins are published on [hex.pm](https://hex.pm), for example [hex.pm/packages/membrane_opus_plugin](https://hex.pm/pakcages/membrane_opus_plugin) and docs are at [hexdocs](https://hexdocs.pm), like [hexdocs.pm/membrane_opus_plugin](https://hexdocs.pm/membrane_opus_plugin). Some plugins require native libraries installed in your OS. Those requirements, along with usage examples are outlined in each plugin's readme.

**Formats**

Apart from plugins, Membrane has stream formats, which live in `membrane_X_format` repositories, where `X` is usually a codec or container, for example, [mebrane_opus_format](https://github.com/membraneframework/mebrane_opus_format). Stream formats are published the same way as packages and are used by elements to define what kind of stream can be sent or received. They also provide utility functions to deal with a given codec/container.

**Core**

[Membrane Core](https://github.com/membraneframework/membrane_core) is the heart and soul of the Membrane Framework. It is written entirely in Elixir and provides the internal mechanisms and API that allow you to prepare processing elements and link them together in a convenient yet reliable way. Note that Membrane Core does not contain any multimedia-specific logic. 
The documentation for the developer's API is available at [hexdocs](https://hexdocs.pm/membrane_core/readme.html).

**Standalone libraries**

Last but not least, Membrane provides tools and libraries that can be used standalone and don't depend on the `membrane_core`, for example, [video_compositor](https://github.com/membraneframework/video_compositor), [ex_sdp](https://github.com/membraneframework/ex_sdp) or [unifex](https://github.com/membraneframework/unifex).

## Where can I learn Membrane?
There are a number of resources available for learning about Membrane:

### This guide
The following sections in that guide will introduce the main concepts of creating Membrane elements and pipelines, without focusing on the specific details of multimedia processing.

### Demos
The [membrane_demo](https://github.com/membraneframework/membrane_demo) repository contains many projects, scripts and livebooks that cover different use cases of the framework. It's a good place to learn by example.

### Tutorials
For a step-by-step guide to implementing a specific system using Membrane, check out our [tutorials](https://membrane.stream/learn).

### Documentation
For more detailed information, you can refer to the Membrane Core documentation and the documentation for the Membrane packages maintained by the Membrane team, both of which can be accessed [here](https://hex.pm/orgs/membraneframework).

If you see something requiring improvement in this guide, feel free to create an issue or open a PR in the [membrane_tutorials](https://github.com/membraneframework/membrane_tutorials) repository.


Get Started with Membrane

# Pipelines

Building `Pipeline`s is the way to create streaming applications with Membrane. Pipeline allows you to spawn `Element`s and establish data flow between them. Pipelines can also communicate with elements or terminate them. Elements within a pipeline are often referred to as its `children` and the pipeline is their `parent`.

Connecting elements together is called `linking`. 

To create a pipeline, you need to implement the [Membrane.Pipeline](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html) behavior. It boils down to implementing callbacks and returning actions from them. For a simple pipeline, it's sufficient to implement the [handle_init](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#c:handle_init/2) callback, which is called upon the pipeline startup, and return the [spec](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html#t:spec/0) action, which spawns and links elements. Let's see it in an example:

## Sample pipeline

```elixir
Mix.install([
  :membrane_hackney_plugin,
  :membrane_mp3_mad_plugin,
  :membrane_portaudio_plugin,
])

defmodule MyPipeline do
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, mp3_url) do
    spec =
      child(%Membrane.Hackney.Source{
        location: mp3_url, hackney_opts: [follow_redirect: true]
      })
      |> child(Membrane.MP3.MAD.Decoder)
      |> child(Membrane.PortAudio.Sink)

    {[spec: spec], %{}}
  end
end

mp3_url = "https://raw.githubusercontent.com/membraneframework/membrane_demo/master/simple_pipeline/sample.mp3"

Membrane.Pipeline.start_link(MyPipeline, mp3_url)
```

The code above is one of the simplest examples of Membrane usage. It plays an MP3 file through your computer's default audio playback device with the help of the [PortAudio](http://www.portaudio.com/) audio I/O library. Let's digest this code and put it to work playing some sound.
> **Elixir**
>
> Membrane is written in Elixir. It's an awesome programming language of the functional paradigm with great fault tolerance and process management, which made it the best choice for Membrane.
> If you're not familiar with it, you can use [this cheatsheet](https://devhints.io/elixir) for a quick look-up.
> We encourage you to also take a [deep look into Elixir](https://elixir-lang.org/getting-started/introduction.html) and learn how to use it to take full advantage of all its awesomeness. We believe you'll fall in love with Elixir too!


You have two options to run the snippet:

- Option 1: Click [here](https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2Fmembraneframework%2Fmembrane_core%2Fblob%2Fmaster%2Fexample.livemd). You'll be directed to install [Livebook](https://livebook.dev), an interactive notebook similar to Jupyter, and it'll open the snippet in there for you. Then just click the 'run' button in there.

- Option 2: If you don't want to use Livebook, you can [install Elixir](https://elixir-lang.org/install.html), type `iex` to run the interactive shell and paste the snippet there.


## Sample pipeline explained

Let's figure out step-by-step what happens in the sample pipeline.

Firstly, we install the needed dependencies. They are plugins, that contain elements that we will use in the pipeline.

```elixir
Mix.install([
  :membrane_hackney_plugin,
  :membrane_mp3_mad_plugin,
  :membrane_portaudio_plugin,
])
```

Instead of creating a script and using `Mix.install`, you can also [create a Mix project](https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html) and add these dependencies to `deps` in `mix.exs` file.

After installing the dependencies, we can create a module for our pipeline:

```elixir
defmodule MyPipeline do
  use Membrane.Pipeline

end
```

Using the `Membrane.Pipeline` behaviour means we are treating our module as a Membrane Pipeline, so we will have access to functions defined in the `Membrane.Pipeline` module, and we can implement some of its callbacks. Let's implement the `handle_init/2` callback, which is a function that is invoked to initialize a pipeline during start-up:

```elixir
defmodule MyPipeline do
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, path_to_mp3) do

  end
end
```

> If the concept of callbacks and behaviours is new to you, you can read more about it [here](https://elixir-lang.org/getting-started/typespecs-and-behaviours.html#behaviours), or see examples of behaviours in the Elixir standard library, for example the [GenServer](https://elixir-lang.org/getting-started/mix-otp/genserver.html) and [Supervisor](https://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html) behaviours.

We'll use `handle_init` to specify all its [elements](../glossary/glossary.md#element) as children and set up links between them to define the order in which data will flow through the pipeline:

```elixir
@impl true
def handle_init(_ctx, path_to_mp3) do
  spec =
    child(%Membrane.Hackney.Source{
      location: mp3_url, hackney_opts: [follow_redirect: true]
    })
    |> child(Membrane.MP3.MAD.Decoder)
    |> child(Membrane.PortAudio.Sink)

  {[spec: spec], %{}}
end
```

The [child](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#child/2) function allows us to spawn particular elements. By piping one child to another with the `|>` operator, we can specify the order of the elements in the pipeline. Thus, the code above links `Membrane.Hackney.Source` to `Membrane.MP3.MAD.Decoder` and `Membrane.MP3.MAD.Decoder` to `Membrane.PortAudio.Sink`. Here's what they are:
- Hackney source - an element based on the [Hackney HTTP library](https://github.com/benoitc/hackney), that downloads a file via HTTP chunk by chunk, and sends these chunks through its `output` pad. We pass two options to it: a URL where the MP3 is stored and a flag to make it follow HTTP redirects.
- MP3 decoder - an element based on [libmad](https://github.com/markjeee/libmad), that accepts MP3 audio on the `input` pad and sends the [decoded](../glossary/glossary.md#decoding) audio through the `output` pad.
- PortAudio sink - an element that accepts decoded audio on its `input` pad and uses the [PortAudio](https://github.com/PortAudio/portaudio) library to play in on the speaker.

In our spec, we don't mention the names of the pads, because `input` and `output` are the defaults. However, we could explicitly specify them:

```elixir
spec =
  child(%Membrane.Hackney.Source{
    location: mp3_url, hackney_opts: [follow_redirect: true]
  })
  |> via_out(:output)
  |> via_in(:input)
  |> child(Membrane.MP3.MAD.Decoder)
  |> via_out(:output)
  |> via_in(:input)
  |> child(Membrane.PortAudio.Sink)
```

Even though not necessary here, [via_in](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_in/3) and [via_out](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_out/3) are useful in more complex scenarios that we'll cover later.

The value returned from `handle_init`:

```elixir
{[spec: spec], %{}}
```

is a tuple containing the list of actions and the state.
- Actions are the way to interact with Membrane. Apart from `spec`, you can for example return `terminate: reason` that will stop the elements and terminate the pipeline. Most actions, including `spec`, can be returned from multiple callbacks, allowing, for example, to spawn elements on demand. Check the [Membrane.Pipeline](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html) behavior for the available callbacks and [Membrane.Pipeline.Action](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html) for the available actions.
- State is an arbitrary data that will be passed to subsequent callbacks as the last argument. It's usually a map. As we have no use for the state in this case, we just set it to an empty map.

When we have created our pipeline module, we can call [Membrane.Pipeline.start_link](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#start_link/3) to run it:

```elixir
Membrane.Pipeline.start_link(MyPipeline, mp3_url)
```

We pass to it the pipeline module and options, which in our case is the `mp3_url`. The options are passed directly to the `handle_init` callback.

Congratulations! You've just built and run your first Membrane application. Let's now have a deeper look at the elements.


Pipelines

# Elements

Elements in Membrane are the most basic entities responsible for processing multimedia.
Each instance of an element is an Elixir process, that has an internal state and communicates by message passing. You've already seen some examples of elements in the previous chapter.

Elements are spawned and controlled by their parent, which can be a pipeline or a bin (we'll cover bins in the subsequent chapter).

## Element types

The basic types of elements are the following:

* `Source` - fetches the stream from outside of the pipeline and delivers it to other elements
* `Sink` - consumes the stream from other elements
* `Filter` - receives the stream from other elements, processes it and sends it further to the subsequent elements
* `Endpoint` - a `Source` and a `Sink` combined - can both deliver and consume the data from other elements

To create an element, you need to `use` the appropriate module - [Membrane.Source](https://hexdocs.pm/membrane_core/Membrane.Source.html), [Membrane.Sink](https://hexdocs.pm/membrane_core/Membrane.Sink.html), [Membrane.Filter](https://hexdocs.pm/membrane_core/Membrane.Filter.html) or [Membrane.Endpoint](https://hexdocs.pm/membrane_core/Membrane.Endpoint.html), for example:

```elixir
defmodule MyElement do
  use Membrane.Filter

  # Element implementation
end
```

The `Element implementation` consists of defining pads, options and callbacks. Let's find out how to do that.

## Pads

As you already learned, pads allow the creation of the flow of data between elements. Pads, much like contact pads on a printed circuit board, are inputs and outputs of an element and are used to connect the elements with one another.
Because of that, there are two types of pads: `input` and `output`. It is worth mentioning that `Source` elements may only contain `output` pads, `Sink` elements contain only `input` pads, and `Filter` and `Endpoint` elements can have both of them.

Every pad should define the format of data that it is expecting. This format can be, for example,
raw audio with a specific sample rate or encoded audio in a given format.

To send data between elements, their pads need to be linked. There are a couple of rules that apply to pad linking:

* One pad of an element can only be linked with one pad from another element.
  (Dynamic pads can help with that limitation; you'll learn about them in the `pads_and_linking` chapter)
* Only links between `output` and `input` pads are allowed.
* Accepted stream formats of pads have to be compatible.

### Defining pads

Pads can be defined using [def_input_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#def_input_pad/2) and [def_output_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithOutputPads.html#def_output_pad/2) macros. They both accept the pad name and the list of properties. The name allows for the identification of the pad. If an element has a single input or output pad, the convention is to name it `input` or `output`, respectively. The pad properties are listed below:

* `accepted_format` - A pattern for a stream format expected on the pad, for example `Membrane.RawAudio` or `%Membrane.RawAudio{channels: 2}`. It serves documentation purposes and is validated in runtime.
* `flow_control` - Configures how back pressure should be handled on the pad. You can choose from the following options:
  * `:auto` - Membrane automatically manages the flow control. It works under the assumption that the element does not need to block or slow down the processing rate, it just processes or consumes the stream as it flows. This option is not available for output pads of `Source` end `Endpoint` elements, while for all other pads it's the default.
  * `:manual` - You need to manually control the flow control by using the `demand` action on `input` pads and implementing the `handle_demand` callback for `output` pads.
  * `:push` - It's a simple mode where an element producing data pushes it right away through the `output` pad. An `input` pad in this mode should be always ready to process that data.
* `demand_unit` - Either `:bytes` or `:buffers`, specifies what unit will be used to request or receive demands. Must be specified for inputs that have `flow_control` is set to `:manual`.
* `availability` - Either `:always` (default) - meaning the pad is static and available from the moment an element is spawned, or `:on_request` meaning it is dynamic. We'll learn more about it in the `Pads and linking` chapter.
* `options` - Optional; specification of options accepted by the pad. We'll learn more about it in the `Pads and linking` chapter.

A pad definition may look like this:

```elixir
def_input_pad :input, flow_control: :auto, accepted_format: %Membrane.RawAudio{channels: 2}
```

It means that the element has a static input pad called `input`, with automatic flow control, that accepts [raw audio](http://hexdocs.pm/membrane_raw_audio_format) with two channels.

## Options

Element options make it possible to pass configuration data to an element. Elements aren't required to accept any options, but it's useful in many cases. Available options can be specified using the [def_options](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#def_options/1) macro, for example:

```elixir
def_options some_option: [
              spec: integer() | string(),
              default: 0,
              description: """
              This option is intended for...
              """
            ],
            other_option: [
              # ...
            ]
```

Each option can have the following fields:
- `spec` - The [typespec](https://hexdocs.pm/elixir/1.12/typespecs.html) of the values that option can have. Defaults to `any`.
- `default` - The value that the option will have if it's not specified. If the default is not provided, the option must be always explicitly specified.
- `description` - Write here what the option does. It will be included in the module documentation.

We'll see a practical example of defining options in the [sample element](#sample-element). 

## Callbacks

Apart from specifying pads and options, creating an element involves implementing callbacks. They have different responsibilities and are called in a specific order. As in the case of pipelines, callbacks interact with the framework by returning [actions](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html). Here are some most useful callbacks:

[handle_init](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_init/2) is invoked once, upon the element creation.
It receives options specified by the user, which should be parsed and on their base,
the element should create and initialize its internal state. It is called synchronously (the parent waits until it returns), thus you shouldn't perform any long tasks there.

[handle_setup](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_setup/2) is invoked right after `handle_init`. It's intended for resource allocation or some potentially time-consuming initialization. If you need to make sure that resources are properly released upon element termination, use [Membrane.ResourceGuard](https://hexdocs.pm/membrane_core/Membrane.ResourceGuard.html) or [Membrane.UtilitySupervisor](https://hexdocs.pm/membrane_core/Membrane.UtilitySupervisor.html)

After `handle_setup`, the following callbacks can be called at any point:
- [handle_pad_added](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_added/3) and [handle_pad_removed](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_removed/3) are called when a dynamic pad is added and removed, respectively
- [handle_parent_notification](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_parent_notification/3) is called whenever the parent sends a notification to the element; elements can send notifications the other way with the **notify** action

[handle_playing](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_playing/2) is called when the stream processing is ready to start. From that point, you can return the following actions:
- [stream_format](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:stream_format/0) tells the subsequent element what kind of stream it should expect on the given pad
- [buffer](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:buffer/0) sends media data to the subsequent element; stream_format has to be sent before the first buffer
- [event](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:event/0) sends a custom struct to the subsequent or preceding element; downstream events are sent in order with buffers
- [demand](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:demand/0) requests data from the previous element; only works for pads in `flow_control: manual` mode
- [end_of_stream](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:end_of_stream/0) tells the subsequent element that the stream has finished, nothing can be sent through that pad afterward

After `handle_playing`, you should expect the following callbacks to be called:

- [handle_stream_format](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_stream_format/4) tells you what kind of stream you should expect on the given pad; called at least once, before `handle_start_of_stream`, may be called later when the stream format changes

- [handle_start_of_stream](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_start_of_stream/3) is called just before the first buffer arrives from the preceding element

- [handle_buffer](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_buffer/4) is called every time a buffer arrives from the preceding element

- [handle_event](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_event/4) is called once an event arrives from the preceding or subsequent element

- [handle_demand](https://hexdocs.pm/membrane_core/Membrane.Element.WithOutputPads.html#c:handle_demand/5) is called when the subsequent element requests data on the given pad; only works for pads in `flow_control: :manual` mode

- [handle_end_of_stream](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_end_of_stream/3) is called when the stream has finished; it may be because the preceding element explicitly returned `end_of_stream` action, the pad is about to be unlinked or the current element is about to terminate

Finally, [handle_terminate_request](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_terminate_request/2) is called when the parent decides to remove the element. By default, it returns the `terminate: :normal` action and the element terminates gracefully. Note that this callback is only called when the element is gracefully asked to terminate.


## Sample element

That's enough for the theory, let's write some code! We'll create a sample element and plug it into the pipeline from the previous chapter. Here's the element:

```elixir
defmodule VolumeKnob do
  @moduledoc """
  Membrane filter that changes the audio volume
  by the gain passed via options.
  """
  use Membrane.Filter

  alias Membrane.RawAudio

  def_input_pad :input, accepted_format: RawAudio, flow_control: :auto
  def_output_pad :output, accepted_format: RawAudio, flow_control: :auto

  def_options gain: [
    spec: float(),
    description: """
    The factor by which the volume will be changed.

    A gain smaller than 1 reduces the volume and gain
    greater than 1 increases it.
    """
  ]

  @impl true
  def handle_init(_ctx, options) do
    {[], %{gain: options.gain}}
  end
  
  @impl true
  def handle_buffer(:input, buffer, ctx, state) do
    stream_format = ctx.pads.input.stream_format
    sample_size = RawAudio.sample_size(stream_format)
    payload =
      for <<sample::binary-size(sample_size) <- buffer.payload>>, into: <<>> do
        value = RawAudio.sample_to_value(sample, stream_format)
        scaled_value = round(value * state.gain)
        RawAudio.value_to_sample(scaled_value, stream_format)
      end

    buffer = %Membrane.Buffer{buffer | payload: payload}
    {[buffer: {:output, buffer}], state}
  end
end
```

As the `moduledoc` says, the element can be used to adjust the audio volume. As we create a filter, we start with `use Membrane.Filter` clause. Then we define pads, one input and one output:

```elixir
alias Membrane.RawAudio

def_input_pad :input, accepted_format: RawAudio, flow_control: :auto
def_output_pad :output, accepted_format: RawAudio, flow_control: :auto
```

The element is going to receive raw audio and send the raw audio too. The raw audio (sometimes referred to as PCM - Pulse Code Modulation) is a simple digital representation of an audio wave, that we can operate on - for example, change the volume. The `Membrane.RawAudio` format is defined in the `membrane_raw_audio_format` package.

Since the element only transforms the stream as it flows, we can safely set `flow_control` to `auto` on both pads.

After defining the pads, we can define options. In this case, it's a single option - `gain` by which the volume will be changed.

```elixir
def_options gain: [
  spec: number(),
  description: """
  The factor by which the volume will be changed.

  A gain smaller than 1 reduces the volume and gain
  greater than 1 increases it.
  """
]
```

It's important to provide the type spec and description for each option so that everyone knows how to use it.

Next, we implement the first callback - `handle_init`:

```elixir
@impl true
def handle_init(_ctx, options) do
  {[], %{gain: options.gain}}
end
```

The callback does not return any actions (thus the empty list), but it saves the gain passed through options in the state.

Then goes the main part of the element - the `handle_buffer` callback:

```elixir
@impl true
def handle_buffer(:input, buffer, ctx, state) do
```

The callback is called whenever a buffer arrives on a pad, and receives four arguments:
- the pad where the buffer arrived,
- the [Membrane.Buffer](https://hexdocs.pm/membrane_core/Membrane.Buffer.html) structure carrying the stream data,
- [Membrane.Element.CallbackContext.t](https://hexdocs.pm/membrane_core/Membrane.Element.CallbackContext.html#t:t/0), providing some useful information about the element,
- the element's state that we created in `handle_init`.

Firstly, we use the callback context to get the stream format present on the pad and use a utility from [Membrane.RawAudio](https://hexdocs.pm/membrane_raw_audio_format/Membrane.RawAudio.html) to calculate the sample size:
```elixir
stream_format = ctx.pads.input.stream_format
sample_size = RawAudio.sample_size(stream_format)
```

We could have implemented the `handle_stream_format` callback and stored the `sample_size` in the element's state too. When there's more work to be done once the stream format arrives, it's the preferred approach, though in a simple case like this we're good using the callback context.

The sample size is the amount of bytes that each audio sample takes. We'll use it to extract each sample from the payload:

```elixir
payload =
  for <<sample::binary-size(sample_size) <- buffer.payload>>, into: <<>> do
```

Now we can convert each sample to an integer with another utility from `Membrane.RawAudio`: [sample_to_value](https://hexdocs.pm/membrane_raw_audio_format/Membrane.RawAudio.html#sample_to_value/2). Having the integer, we can multiply it by the gain and convert it back to the binary representation.

```elixir
    value = RawAudio.sample_to_value(sample, stream_format)
    scaled_value = round(value * state.gain)
    RawAudio.value_to_sample(scaled_value, stream_format)
  end
```

Finally, we can update the payload and forward the buffer to the output pad using the [buffer](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:buffer/0) action.

```elixir
  buffer = %Membrane.Buffer{buffer | payload: payload}
  {[buffer: {:output, buffer}], state}
end
```

Let's test our element by plugging it into the pipeline from the previous chapter. Since it accepts `Membrane.RawAudio`, we should plug it after the decoder, which accepts encoded audio and outputs raw audio. Let's update the spec in the `handle_init` callback of our pipeline:

```elixir
spec =
  child(%Membrane.Hackney.Source{
    location: mp3_url, hackney_opts: [follow_redirect: true]
  })
  |> child(Membrane.MP3.MAD.Decoder)
  |> child(%VolumeKnob{gain: 0.2})
  |> child(Membrane.PortAudio.Sink)
```

Let's run the pipeline again:

```elixir
Membrane.Pipeline.start_link(MyPipeline, mp3_url)
```

Since we set the gain to `0.2`, the audio should play quieter than before.

In this chapter, you learned how elements work and how to create one. Now let's figure out what are `Bin`s.


Elements

# Bins

Bins, similarly to pipelines, are containers for elements. However, at the same time, they can be placed and linked within pipelines. Although a bin is a separate Membrane entity, it can be perceived as a pipeline within an element. Bins can also be nested within one another, though we don't recommend too much nesting, as it may end up hard to maintain. For an example of a bin, have a look at the [RTMP source bin](https://github.com/membraneframework/membrane_rtmp_plugin/blob/master/lib/membrane_rtmp_plugin/rtmp/source/bin.ex) or [HTTP adaptive stream sink bin](https://github.com/membraneframework/membrane_http_adaptive_stream_plugin/blob/master/lib/membrane_http_adaptive_stream/sink_bin.ex).

The main use cases for a bin are:
- creating reusable element groups,
- encapsulating children's management logic, for instance, dynamically spawning or replacing elements as the stream changes.

## Bin's pads

Bin's pads are defined and linked similarly to element's pads. However, their role is limited to proxy the stream to other elements and bins inside (inputs) or outside (outputs). To achieve that, each input pad of a bin needs to be linked to both an output pad from the outside of a bin and an input pad of its child inside. Accordingly, each bin's output should be linked to output inside and input outside of the bin.

## Bin and the stream

Although the bin passes the stream through its pads, it does not access it directly, so callbacks such as `handle_process` or `handle_event` are not found there. This is because the responsibility of the bin is to manage its children, not to process the stream. Whatever the bin needs to know about the stream, it should get through notifications from the children.

## Bin as a black box

Bins are designed to take as much responsibility for their children as possible so that pipelines (or parent bins) don't have to depend on bins' internals. That's why notifications from the children are sent to their direct parents only. Also, messages received by a bin or pipeline can be forwarded only to its direct children.


Bins

# Pads and linking

You learned how to link elements in the `Pipeline` chapter and how to define pads in the `Element` chapter. Now, let's have a deeper dive into pads and their capabilities.

## Dynamic Pads

A dynamic pad is a type of pad that acts as a template - each time some other pad is linked to a dynamic pad, a new instance of it is created.

Dynamic pads don't have to be linked when the element is started. This needs to be handled by the element, but in return, it gives new possibilities when the number of pads can change on the fly.

Another use case for dynamic pads is when the number of pads is not known at the compile time.
For example, an audio mixer may have any number of inputs.


### Creating an element with dynamic pads

To make a pad dynamic, you need to set its `availability` to `:on_request` in [def_input_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#def_input_pad/2) or [def_output_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithOutputPads.html#def_output_pad/2).

Now, each time some element is linked to this pad, a new instance of the pad is created and [handle_pad_added](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_added/3) callback is invoked. Instances of a dynamic pad can be referenced as `Pad.ref(pad_name, pad_id)`. When a dynamic pad is unlinked, the [handle_pad_removed](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_removed/3) callback is called.

### Gotchas

As usual, with great power comes great responsibility. When implementing an element with dynamic pads, you need to consider what should happen when they are added or removed. Usually, you need to implement `handle_pad_added` and `handle_pad_removed` callbacks, and the logic of an element may generally become more complicated.

### Linking dynamic pads

Let's see how to link dynamic pads. We'll use [membrane_file_plugin](https://github.com/membraneframework/membrane_file_plugin), a plugin that allows reading and writing to files, and [membrane_tee_plugin](https://github.com/membraneframework/membrane_tee_plugin) which allows forwarding the stream from a single input to multiple outputs. Running the pipeline below with `Membrane.Pipeline.start_link(MyPipeline)` should copy the "source" file to "target1", "target2" and "target3" files. Don't forget to create the "source" file before.

```elixir
Mix.install([
  :membrane_file_plugin,
  :membrane_tee_plugin
])

defmodule MyPipeline do
  use Membrane.Pipeline

  alias Membrane.{File, Tee}

  @impl true
  def handle_init(_ctx, _options)
    spec = [
      child(%File.Source{location: "source"})
      |> child(:tee, Tee.Parallel),
      get_child(:tee) |> child(%File.Sink{location: "target1"}),
      get_child(:tee) |> child(%File.Sink{location: "target2"}),
      get_child(:tee) |> child(%File.Sink{location: "target3"})
    ]
  end
end
```

The [Membrane.Tee.Parallel](https://hexdocs.pm/membrane_tee_plugin/Membrane.Tee.Parallel.html) element has a single static input and a single dynamic output pad. Because the output is dynamic, each time we link it, a new pad instance is created with a unique reference. In the example above, pad references were generated automatically. It's possible to specify them directly with [via_in](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_in/3) or [via_out](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_out/3) and [Membrane.Pad.ref](https://hexdocs.pm/membrane_core/Membrane.Pad.html#ref/1):

```elixir
spec = [
  child(%File.Source{location: "source"})
  |> child(:tee, Tee.Parallel),
  get_child(:tee) |> via_out(Pad.ref(:output, 1)) |> child(%File.Sink{location: "target1"}),
  get_child(:tee) |> via_out(Pad.ref(:output, 2)) |> child(%File.Sink{location: "target2"}),
  get_child(:tee) |> via_out(Pad.ref(:output, 3)) |> child(%File.Sink{location: "target3"})
]
```

In this case, it won't make a difference, but elements can rely on pad references and use them to identify the stream that should be sent or received through the given pad.

## Pad options

Just like elements, pads can have options. They have to be defined with the `options` key in `def_input_pad` and `def_output_pad`, using the same syntax as the element options. For example, [Membrane.AudioMixer](https://hexdocs.pm/membrane_audio_mix_plugin/Membrane.AudioMixer.html) does it to make its `input` pad accept the `offset` option:

```elixir
def_input_pad :input,
    availability: :on_request,
    options: [
      offset: [
        spec: Time.non_neg(),
        default: 0,
        description: "Offset of the input audio at the pad."
      ]
    ],
    # ...
```

Pad options can be set via a keyword list within the second argument of `via_in` or `via_out`. Here's how to provide the `offset` option to the mixer:

```elixir
spec = [
  # ...
  child(Membrane.MP3.MAD.Decoder)
  |> via_in(:input, options: [offset: Membrane.Time.seconds(2)])
  |> child(Membrane.AudioMixer)
  # ...
]
```

Pad options' values can be accessed in the context of the `handle_pad_added` callback:

```elixir
@impl true
def handle_pad_added(pad, context, state) do
  %{offset: offset} = context.pad_options
  # ...
end
```


Pads &amp; linking

# Flow Control

Once we have linked the elements together, there comes a moment when they can start sending buffers through the pads. However, how to control the flow of buffers between elements?

Membrane solves this problem using its own backpressure mechanism and demands.

## Types of `flow_control`

When defining a pad od an element, you can pass one of 3 values in the `flow_control` field: `auto`, `push`, or `manual`. They determine how the backpressure mechanism works on the pad they relate to.

- Input pad with `flow_control: :push` - an element with such a pad must always be ready to receive a new buffer on that pad. The output pad linked with such an input pad must also have `flow control` set to `:push`. It is not advised to use `flow_control: :push` in input pads, as it makes them prone to overflow.

- Output pad with `flow_control: :push` - the element decides for itself when to send how many buffers via a given pad.

- Input pad with `flow_control: :manual` - the element directly, using the `demand` action, requests how much data it wants to receive from this pad. Membrane guarantees that more data will never come to the input pad than was demanded.

- Output pad with `flow_control: :manual` - the element gets the information, through the `c:Membrane.Element.WithOutputPads.handle_demand/5` callback, how much data a specific output pad needs. An element having an output pad with `flow_control: :manual` is obligated to satisfy the demand on it.

- Input pad with `flow_control: :auto` - unlike in manual demands, the element doesn't have to directly specify the demand value on the input pad to receive buffers on it. In `:auto` `flow control`, the framework will manage the demand value itself. In the case of Sources and Endpoints, it will make sure that the demand on the input pad is positive as long as the number of buffers waiting for processing in the mailbox is not too large. In the case of filters, Membrane will take care of a positive demand under one additional condition - the demand value on all output pads with `flow_control: :auto` of this element must also be positive.

- Output pad with `flow_control: :auto` - an auto output pad does not require from the element to implement the `handle_demand` callback. The demand value on this pad only matters when the framework itself calculates the value of the auto demand on the input pads. The only type of Membrane Element that can have output pads with `flow_control: :auto` is a `Membrane.Filter`. 

`flow_control: :manual` is useful in cases, when element wants to get a specific amount of data on each pad (e.g. to satisfy demand on 1 buffer on pad `:output`, element need 1 buffer from pad `:input_a`, 10 buffers from pad `:input_b` and 100 bytes from pad `:input_c`). Comparing to `flow_control: :auto`, `flow_control: :manual` can solve a broader class of problems, but, on the other hand, it requires more code in your element and is less performant.

`flow_control: :auto` is the best solution, when you want to provide a backpressure mechanism, but your element has only one input pad or it doesn't care about the proportion of the amount of data incoming on each input pad. A good example might be a pipeline made of filters linked one after the other, with a sink or a source on each end. In such a case, using `flow_control: :auto` in filters would ensure, that the pipeline would adapt its processing pace to the slowest element, without the necessity to implement `c:Membrane.Element.WithOutputPads.handle_demand/5` in every element. Additionally, the whole pipeline will be faster, because of the lack of performance overhead caused by `flow_control: :manual`.

## Linking pads with different `flow_control`

Two pads with the same type of `flow_control` can always be linked together. `flow_control: auto` can also always be linked with `flow_control: manual`. However, the situation becomes complicated when we try to link two pads with different types of `flow_control`, where one of them has `flow_control: push`.

When the input pad has `flow_control: push`, the output pad linked to it must also have `flow_control: push`. An attempt to link different type of an output pad will cause an error to be thrown.

Output pads with `flow_control: push` can be linked with input pads having `flow control` `:auto` or `:manual`. The demand value on such an input pad does not affect the behavior of the element with the push output pad.

When an element with a push output pad sends amount of data, that exceeds demands on the input pad, when input pad `flow_control` is: 
- `manual`, these buffers will wait in line until the element raises the demand on its input by returning the `demand` action.
- `auto`, these buffers will be processed, and the demand value on the input pad will drop below zero.

```elixir
get_child(:mixer)
|> via_in(:input, toilet_capacity: 500)
|> get_child(:sink)
```

Flow control

# Observability & logging

To observe and debug Membrane, you can use a variety of tools provided by the BEAM, like [observer](https://www.erlang.org/doc/man/observer.html) or [Process.info](https://hexdocs.pm/elixir/1.15.5/Process.html#info/1). Apart from them, Membrane provides dedicated utilities for observability and logging.

## Introspecting pipelines with kino_membrane

[kino_membrane](https://github.com/membraneframework/kino_membrane) allows for introspecting Membrane pipelines in [Livebook](https://livebook.dev). You can either run the pipeline inside livebook ([see example](https://hexdocs.pm/kino_membrane/pipeline_in_livebook.html)) or connect to a running VM ([see example](https://hexdocs.pm/kino_membrane/connect_to_node.html)). Then, it's enough to install kino_membrane:

```elixir
Mix.install([:kino_membrane])
```

and run

```elixir
KinoMembrane.pipeline_dashboard(pipeline_pid)
```

in Livebook, and you'll get a panel with the pipeline graph, metrics and other utilities:

![Kino Membrane](assets/images/kino_membrane.png)

Check [kino_membrane](https://github.com/membraneframework/kino_membrane) for details.

## Logging

Apart from the usual `Logger` configuration, logging in Membrane can be additionally configured, also via Elixir's `Config`. It allows to enable verbose mode and customize metadata, for example:

```elixir
import Config
config :membrane_core, :logger, verbose: true
```

For this change to take effect, it may be necessary to recompile the whole project with `mix deps.compile --force && mix compile --force`. See [`Membrane.Logger`](https://hexdocs.pm/membrane_core/Membrane.Logger.html) for details.

Moreover, in a pipeline or bin, you can add logger metadata to the [children spec](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html):

```elixir
@impl true
def handle_init(_ctx, opts) do
  spec = {
    child(Some.Element) |> child(Some.Bin),
    log_metadata: [pipeline_id: opts.id]
  }

  {[spec: spec], state}
end
```

and it will add the `pipeline_id` metadata key to all the logs coming from `Some.Element` and `Some.Bin` and all its descendants.

To have the metadata displayed in the logs, you need to configure it as well:
```elixir
import Config
config :logger, :console, metadata: [:pipeline_id]
```


Observability &amp; logging

# Native code integration

In Membrane we try to make use of Elixir goodness as often as possible. However, sometimes it is necessary to write some native code, for example to integrate with existing C libraries. To achieve that, we use _natives_, which are either [NIFs](http://erlang.org/doc/man/erl_nif.html) or [C nodes](http://erlang.org/doc/man/ei_connect.html). Both of them have their tradeoffs:
- NIFs don't introduce significant overhead, but in case of failure they can even crash entire VM,
- CNodes are isolated and can be supervised like Elixir processes, but they are slower due to the inter-process communication.

That's why we use them only when necessary and created some tools to make dealing with them easier. For interoperability with C and C++, we use [Bundlex](https://github.com/membraneframework/bundlex) and [Unifex](https://github.com/membraneframework/unifex). For calling Rust, we use [Rustler](https://github.com/rusterlium/rustler).

## [Bundlex](https://github.com/membraneframework/bundlex)

To simplify and unify the process of writing and compiling natives, we use our own build tool - [Bundlex](https://github.com/membraneframework/bundlex). It is a multi-platform tool that provides a convenient way of compiling and accessing natives from Elixir code.
For more information, see [Bundlex's GitHub page](https://github.com/membraneframework/bundlex).

## [Unifex](https://github.com/membraneframework/unifex)

Process of creating natives is not only difficult but also quite arduous, because it requires using cumbersome Erlang APIs, and thus a lot of boilerplate code. To make it more pleasant, we created [Unifex](https://github.com/membraneframework/unifex), a tool that is responsible for generating interfaces between C or C++ libraries and Elixir on the base of short `.exs` configuration files. An important feature of Unifex is that you can write the C/C++ code once and use it either as a NIF or as a C node.

A quick introduction to Unifex is available [here](https://hexdocs.pm/unifex/creating_unifex_natives.html).



Native code integration

See how pipeline works in Membrane and learn how to set it up for you

In this tutorial, we will create our very own Membrane Framework consisting of our custom which will fulfill the multimedia processing task.
Despite the fact that the multimedia processing task we will be facing will be really simple, we will need to deal with some problems occurring in real-life scenarios.
At the same time, we will make ourselves comfortable with some concepts of multimedia streaming as well as get in touch with the nomenclature used in this field.
Prepare for an adventure!

## Environment preparation

In order to be able to proceed with the tutorial, you need to have Elixir installed: [How to install Elixir](https://elixir-lang.org/install.html).
We assume, that you are at least slightly familiar with that language - if that is not true, we would like to strongly encourage you to take a look at the [official Elixir tutorial](https://elixir-lang.org/getting-started/introduction.html).
Once you are ready with the Elixir, you can get the project template we have prepared for you:

```console
git clone https://github.com/membraneframework/membrane_basic_pipeline_tutorial
cd membrane_basic_pipeline_tutorial
git checkout template/start
mix deps.get
```

As you can see, the template's code is put on the `template/start` branch of the repository.
In the repository, there is also a `template/end` branch, where you can find the completed template.
If you find yourself lost during the tutorial feel free to check the implementation proposed by us, put on this branch.

## Use Case

Imagine that there is a conversation occurring between two peers and the chat is based on a question-answer scheme.
Each of the peers sends its part of the conversation (that means - the first peer sends questions and the second peers sends the answers to them). We refer to each sentence sent by the peers as a [**frame**](../glossary/glossary.md#frame).
However, due to network limitations each frame sent by the peers is fragmented since only a few chars can be sent at the same time - that means, a single frame is sent as a sequence of [**packets**](../glossary/glossary.md#packet).
In a real network there is a parameter called MTU (*Maximum Transmission Unit*) which is the maximum number of bytes which can be send in a single packet. For the Ethernet II MTU is equal to 1500 bytes. We wanted to simulate the situation,
in which the data needs to be fragmented in order to be transmitted - but we decided to choose a small MTU so that this fragmentation is easy to be spotted.
Each packet consists of the *header* (describing where the particular bunch of characters it transports should be put) and the *body* - aforementioned  bunch of characters.
Below you can see an exemplary frame sent by one peer to the other. It gets fragmented into multiple packets, which later on are sent via the network (during that process their order will probably get messed). On the second peer's side, the packets get assembled into the original frame.
![Example Chat](assets/images/example_chat.drawio.png)

### Packet format

Here is how each packet looks like:

```elixir
[seq:<sequence_id>][frameid:<frame_id>][timestamp:<timestamp>]<text>
```

where:

- sequence_id - the ordering number of the packet (relative to each of the peers). It reflects the position of a particular packet in the packets' stream.
- frame_id - the identifier which consists of the number of the frame to which the body of a given packet belongs, optionally followed by a single **'e'** character (meaning that the packet is the **e**nding packet of the frame). Note that frames are numbered relatively to each peer in that conversation and that frame_id does not describe the global order of the frames in the final file.
- timestamp - a number indicating a time at which a given sentence was said. Timestamp describes the order of the frames from both peers.
- text - the proper body of the packet, in our case - a bunch of characters which could be sent in a single packet.

### Packets generator

We have equipped you with the tool which produces the packets in the format described previously, based on the input conversation. You can use it as a [mix](../glossary/glossary.md#mix) task, by typing:

```console
mix generate_input --packetsPerFrame <packets_per_frame> <input_file_path>
```

where:

- *input_file_path* is a path to the file which contains the text of the conversation. The input file consists of multiple lines of the following format:

```
<speaker id>: <text>
```
<br><br>

- *packets_per_frame* is a number of packets on which each frame will be dismantled

Based on the input file content, that command will create multiple files, `<input_file_path>.<speaker id>.<input file format>` . For instance: `input.txt` with input from speakers `A` and `B` will be split into `input.A.txt` and `input.B.txt` files.
The first file will contain the shuffled list of packets made from the `A` speaker's lines from the input file and the second file will contain a shuffled list of packets made out of the `B` speaker's lines of the input file. That `shuffle` is a way to simulate the imperfectness of the network - in real-life scenario, the order in which the packets are received is not always the same as the order in which they were sent. [Here](https://www.openmymind.net/How-Unreliable-Is-UDP/) you can read about this phenomenon occurring while using [UDP](../glossary/glossary.md#udp).
Below you can see the steps which are taken during the input files generation:<br>
![Example Input](assets/images/example_input.drawio.png)

Create a file with your own input conversation occurring between two speakers or use the `input.txt` file where we have provided you such a conversation. Generate the files containing packets, with the use of `mix generate_input` task.

## Task description

By the end of this chapter you should have generated the files containing packets (i.e. `input.A.txt` and `input.B.txt`).

Your task is to assemble the conversation together from those packets. In the first step you will reproduce the sentences sent by each of the peers by gathering the words sent in packets in the appropriate order. In multimedia processing, we would say that you need to reproduce the *frame* out of the *transport layer packets*. Later on you need to put those *frames* in the correct order, producing the *raw data stream*.
The resulting file should be the same as the input file from which you have created two packets list with the use of the `InputFilesGenerator`.


Introduction

# System Architecture

Once we know what should be done, let's start thinking how should our system look like!

First, let's get familiar with some terms. In the Membrane Framework, there is a concept of the [`pipeline`](../glossary/glossary.md#pipeline) which consists of multiple [`elements`](../glossary/glossary.md#element). Elements are linked with the use of the [`pads`](../glossary/glossary.md#pad), which can be input pads or output pads.
Depending on what type of pads the particular element is equipped with, we distinguish the following types of elements:

- Source - element with only output pads, the first element of each pipeline. It is responsible for fetching the data and transmitting it through the output pad.
- Filter - element with both the input pads and the output pads. Its purpose is to get the data via the input pad, transform it and finally transmit the processed data via the output pad.
- Sink - element with only input pads, the last element of the pipeline. It can be responsible for storing or playing the data received via the input pad.
  Between the pads, the [`buffers`](../glossary/glossary.md#buffer) are being sent.
  Buffers sent between the pads should be of the same type. In the Membrane Framework nomenclature, we say, that the pads should have corresponding capabilities - [`caps`](../glossary/glossary.md#caps).
  One might wonder when does the event of the buffers being sent occurs - well, it depends on the pad's mode!
  If the pad works in the *pull* mode, the buffer is sent when it is demanded by the succeeding element in the pipeline.
  Otherwise, the pad works in the *push* mode, which means that it is pushing the buffers once they are ready, independently from the fact if the following elements want (and are capable of processing) it.
  In our pipeline, all the pads will work in the *pull* mode, which inducts a flow control by a [backpressure](https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7) mechanism.

## Scheme

![Pipeline scheme](assets/images/basic_pipeline.png)

As you can see, our pipeline will consist of two twin branches, one per each of the peers. The branches will be merged with the `Mixer` element and the result produced by this element will be put in the file with the `Sink` elements.
Here you can find the description of the particular elements of the system.

## Elements description

- **Source** - that element is responsible for reading the list of [packets](../glossary/glossary.md#packet) from the file.
- **Ordering Buffer** - this element is responsible for reordering the packets based on their sequence id. The most important part of this element is the buffer, within which there is a place for all the packets, identified by the packet's sequence id. Once the new packet is received, it is placed in the proper place in the buffer, depending on the sequence id. If there is a consistent part of packets "at the bottom of the buffer" (which means - packets with the subsequent sequence ids, which haven't already been sent yet), then this part is sent via the output pad. That is how we make sure that the next element will receive elements sorted by the sequence id.
  Below you can see how the Ordering Buffer is expected to work, once the message "How are you doing?" will be sent in the four packets, each with one word, and received in the following order: (are, you, how, doing?) <br>
  ![Ordering Buffer](assets/images/ordering_buffer.drawio.png) <br>
  
- **Depayloader** - responsible for assembling the packets into the [frames](../glossary/glossary.md#frame). Since the depayloader receives the packets in the order of their sequence id (which means - the order in which they were 'sent'), it is capable of separating out the particular frames, basing on the **e** (ending packet of the frame) characters at the end of the frame id. This element also reads the timestamp from the packets' headers and puts it into buffer's 'pts' (*Presentation timestamp*) field.
- **Mixer** - responsible for mixing the frames received via two input pads. The order in which the frames are mixed is based on their 'pts' (*Presentation timestamp*).
- **Sink** - this element is responsible for writing the received frames to the file. Since the Mixer takes care of sorting the frames based on the timestamp, Sink can take advantage of the fact that the frames will be ordered and write them to the file in the order they are received.

Now, that you have some high-level understanding of the system we can get down to implementation.


System Architecture

# Source

Let's get to the code!
We will start where all the [pipelines](../glossary/glossary.md#pipeline) start - with the `Source` [element](../glossary/glossary.md#element).
Since this will be the first element we implement, we need to find out something more about how the Membrane Framework's elements should be implemented and some concepts associated with them.
The first thing you need to be aware of is that `Membrane. Element` describes a specific behavior, based on the OTP [GenServer's](https://elixir-lang.org/getting-started/mix-otp/genserver.html) behavior.
Our process keeps a state which is updated in callbacks.
We only need to provide an implementation of some callbacks in order to make our element act in the desired way.
The set of callbacks that can be implemented depends on the type of the elements and we will get familiar with them during the implementation of these elements.
However, each callback is required to return a tuple of a [specific form](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#t:callback_return/0).
As you can see, we are returning an optional list of [actions](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html#t:t/0) to be performed, and
the updated state (which later on will be passed to the next invoked callback).
Take your time and read about the possible actions which can be requested to be performed while returning from the callback. Their usage is crucial for the pipeline to work.

As you can judge based on the structure of the project, all the elements will be put in the `lib/elements` directory. Therefore there is a place where `source.ex` with the `Basic.Elements.Source` module's definition should be placed.

## What makes our module a Membrane Framework's element?

Let's start with specifying that our module will implement the `Membrane.Source` behavior as well as alias the modules which will be used later in the module's code:

**_`lib/elements/source.ex`_**

```elixir
defmodule Basic.Elements.Source do
 use Membrane.Source
 alias Membrane.Buffer
 alias Basic.Formats.Packet
 ...
end
```

## Pads and options

Later on, we will make use of [macros](https://elixir-lang.org/getting-started/meta/macros.html) defined in the `Membrane.Source` module:

**_`lib/elements/source.ex`_**

```elixir
defmodule Basic.Elements.Source do
 ...
  def_options location: [
                spec: String.t(),
                description: "Path to the file"
              ]

  def_output_pad :output,
    accepted_format: %Packet{type: :custom_packets},
    flow_control: :manual
 ...
end
```

The first macro, `def_options` allows us to define the parameters which are expected to be passed while instantiating the element. The parameters will be passed as an automatically generated structure `%Basic.Elements.Source{}`. In our case, we will have a `:location` field inside of that structure. This parameter is about to be a path to the files which will contain input [packets](../glossary/glossary.md#packet).
Later on, while instantiating the Source element, we will be able to write:

```elixir
%Basic.Elements.Source{location: "input.A.txt"}
```

and the `:location` option will be passed during the construction of the element.

The second macro, `def_output_pad`, lets us define the output pad. The pad name will be `:output` (which is a default name for the output pad). The second argument of the macro describes the `:accepted_format` - which is the type of data sent through the pad. As the code states, we want to send data in `Basic.Formats.Packet` format.
What's more, we have specified that the `:output` pad will work in `:manual` mode.
You can read more on pad specification [here](https://hexdocs.pm/membrane_core/Membrane.Pad.html#types).

## Initialization of the element

Let's define our first callback! Why not start with [`handle_init/2`](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_init/2), which gets called once the element is created?

**_`lib/elements/source.ex`_**

```elixir
defmodule Basic.Elements.Source do
 ...
  @impl true
  def handle_init(_context, options) do
    {[],
     %{
       location: options.location,
       content: nil
     }}
  end
 ...
end
```

As said before, `handle_init/2` expects a structure with the previously defined parameters to be passed as an argument.
All we need to do there is to initialize the state - our state will be in a form of a map, and for now on we will put there a `location` (a path to the input file) and the `content`, where we will be holding packets read from the file, which haven't been sent yet. The recommended approach is to keep `handle_init/2` lean and defer any heavier initialization to `handle_setup/2` which runs automatically afterwards, if defined. In the next section we'll talk about more involved initialization and resources managed by our element.

> ### TIP
>
> You might also wonder what is the purpose of the `@impl true` specifier, put just above the function signature - this is simply a way to tell the compiler
> that the function defined below is about to be a callback. If we have misspelled the function name (or provided a wrong arguments list), we will be informed in the compilation time.

## Preparing our element

In the `handle_setup/2` callback, we'd like to open, read and save the contents of the input file. We then save it in our state as `content`.

**_`lib/elements/source.ex`_**

```elixir
defmodule Basic.Elements.Source do
 ...
  @impl true
  def handle_setup(_context, state) do
    content =
      File.read!(state.location)
      |> String.split("\n")

    new_state = %{state | content: content}
    {[], new_state}
  end
 ...
end
```

When the setup is complete, the element goes into `:playing` state. It can then demand buffers from previous elements and send its `:stream_format` to subsequent elements. Since we are implementing a sink we do not have any previous element to demand from, but we can specify the format. We can do this, for example, in `handle_playing/2`:

**_`lib/elements/source.ex`_**

```elixir
defmodule Basic.Elements.Source do
 ...
  @impl true
  def handle_playing(_context, state) do
    {[stream_format: {:output, %Packet{type: :custom_packets}}], state}
  end
 ...
end
```

The `:stream_format` action means that we want to transmit the information about the supported [formats](../glossary/glossary.md#stream-format-formerly-caps) through the `output` pad, to the next element in the pipeline. In [chapter 4](../basic_pipeline/04_StreamFormat.md) you will find out more about stream formats and learn why it is required to do so.

## Demands

Before going any further let's stop for a moment and talk about the demands. Do you remember, that the `:output` pad is working in `:manual` mode? That means that the succeeding element has to ask the Source element for the data to be sent and our element has to take care of keeping that data in some kind of buffer until it is requested.
Once the succeeding element requests for the data, the `handle_demand/4` callback will be invoked - therefore it would be good for us to define it:

**_`lib/elements/source.ex`_**

```elixir
defmodule Basic.Elements.Source do
 ...

  @impl true
  def handle_demand(:output, _size, :buffers, _context, state) do
    if state.content == [] do
      {[end_of_stream: :output], state}
    else
      [first_packet | rest] = state.content
      new_state = %{state | content: rest}

      actions = [
        buffer: {:output, %Buffer{payload: first_packet}},
        redemand: :output
      ]

      {actions, new_state}
    end
  end
 ...
end
```

The callback's body describes the situation in which some buffers were requested. Then we are checking if we have any packets left in the list persisting in the state of the element. If that list is empty, we are sending an `end_of_stream` action, indicating that there will be no more buffers sent through the `:output` pad and that is why there is no point in requesting more buffers. 

However, in case of the `content` list of packets being non-empty, we are taking the head of that list, and storing the remaining tail of the list in the state of the element. Later on, we are defining the actions we want to take - that is, we want to return a buffer with the head packet from the original list. We make use of the [`buffer:` action](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:buffer/0), and specify that we want to transmit the [`%Buffer`](https://hexdocs.pm/membrane_core/Membrane.Buffer.html#t:t/0) structure through the `:output` pad. Note the fields available in the `%Buffer` structure - in our case, we make use of only the `:payload` field, which, according to the documentation, can be of `any` type - however, in almost all cases you will need to send binary data within this field. Any structured data (just like timestamps etc.) should be passed in the other fields available in the `%Buffer`, designed especially for that cases. 

Then there's the other action that is taken - the `:redemand` action, queued to take place on the `:output` pad. This action will simply invoke the `handle_demand/4` callback once again, which is helpful when the whole demand cannot be completely fulfilled in the single `handle_demand` invocation we are just processing. The great thing here is that the `size` of the demand will be automatically determined by the element and we do not need to specify it anyhow. Redemanding, in the context of sources, helps us simplify the logic of the `handle_demand` callback since all we need to do in that callback is to supply a single piece of data and in case this is not enough, take a [`:redemand`](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:redemand/0) action and invoke that callback once again. As you will see later, the process of redemanding is even more powerful in the context of [filter elements](../glossary/glossary.md#filter). 

> ### TIP
>
> Membrane also supports `:auto` demand mode, which should cover 90% of use cases. 
> 
> You can learn more about demand modes [here](https://membrane.stream/learn/get_started_with_membrane/6).

In the next chapter we will explore what stream formats are in Membrane.


Source

# Stream format

We owe you something...and we would like to pay it back as soon as possible!
As promised in the [3rd chapter](03_Source.md), we will talk more about the concept of stream formats - which in fact we have used in the previous chapter, but which weren't described sufficiently.

## Why formats are important

Specifying stream formats allows us to define what kind of data is flowing through the [pad](../glossary/glossary.md#pad). 
This isn't necessarily limited to data formats, as you'll see below.
In the Membrane Framework's nomenclature, we say, that we define a stream format specification for a given [element](../glossary/glossary.md#element).

We believe that an example might speak here louder than a plain definition, so we will try to describe it with a real-life scenario example.
Let's say that we are connecting two elements that process the video multimedia.
The link is made between the pads which are working on raw video data.
Here is where stream formats come up - they can be defined with the following constraints:

- data format - in our case, we are having a raw video format
- some additional constraints - i.e. [frame](../glossary/glossary.md#frame) resolution (480p) , framerate (30 fps) etc.

The `:stream_format` action helps us find out if the given elements are capable to communicate with each other. Not only can we not send the data between the pads if the format they are expecting is different - we need to take into consideration some other constraints! We can think of a situation in which the _data format_ would be the same (i.e. raw video data), but the element which receives the data performs a much more complex computation on that data than the sender, and therefore cannot digest such a great amount of data as the sender is capable of transmitting. Then their stream formats wouldn't be compatible, which could be expressed by adding some constraint, i.e. framerate.

Stream formats help us define a contract between elements and prevent us from connecting incompatible elements. That is why it is always better to define precise constraints rather than using `stream_format: :any`.

## When are stream formats compatible?

A comparison between formats is made when an input pad receives the `stream_format` action, and checks whether it matches its [accepted format](https://hexdocs.pm/membrane_core/Membrane.Pad.html#t:accepted_format/0).
Here is how you define a stream format specification:

1. First you need to specify the format module

```elixir
defmodule Formats.Raw do
  defstruct [:pixel_format, :framerate, :width, :height]
end
```

Module name defines the type of the format, however it is possible to pass some other options in a form of a struct. That is why we have defined a structure with the use of `defstruct`. Our format will be described with the following options:

- :pixel_format - pixel format, i.e. [I420](https://en.wikipedia.org/wiki/Chroma_subsampling) ([YUV](https://en.wikipedia.org/wiki/YUV)) or RGB888
- :framerate - number of frames per second, i.e. 30 (FPS)
- :width - width of the picture in pixels, i.e. 480 (px)
- :height - height of the picture in pixels, i.e. 300 (px)

2. We specify the pad of the element with the format we have just defined, using the `:accepted_format` option. For the purpose of an example, let it be the `:input` pad:

```elixir
  def_input_pad :input,
  demand_unit: :buffers,
  accepted_format:
    %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480, height: 300}
    when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60
```
As you can see, the argument of that option is simply a match pattern. The incoming stream format is later confronted against that match pattern. If it does not match, an exception is thrown at the runtime. 

To simplify the pattern definition, there is `any_of/1` helper function that allows to define a alternative of match patterns - the matching will succeed if the stream format received on the pad matches any of the patterns listed as `any_of/1` argument. Below you can see an example of defining alternative of match patterns:

```elixir
def_input_pad :input,
  demand_unit: :buffers,
    accepted_format:
      any_of([
        %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480, height: 300}
        when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60,
        %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 720, height: 480}
        when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60
      ])
```

Our journey with stream formats does not end here. We know how to describe their specification...but we also need to make our elements send the `:stream_format` events so that the following elements will be aware of what type of data our element is producing!

An element can send a stream format as one of the [actions](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html) it can take - the [`:stream_format` action](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:stream_format/0).

Another thing is that we can specify the behavior of an element when it receives the stream format with the use of [`handle_stream_format/4` callback](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_stream_format/4).

For all the [filter elements](../glossary/glossary.md#filter), `handle_stream_format/4` has a default implementation, which is relaying the received format on all the output pads of that filter.
However, if your filter is changing the format of data being sent, it should override the implementation of that callback to prevent formats flying through it, and send the proper spec via the output pads.

For the [source element](../glossary/glossary.md#source), it is necessary to send the format as in each [pipeline](../glossary/glossary.md#pipeline) the source is the first element - formats wouldn't flow through the pipeline if the source element wouldn't have sent them. Sending can be done in the `handle_playing/2` callback.

## Example

Imagine a pipeline, which starts with the source producing a video, which is then passed to the filter, responsible for reducing the quality of that video if it is too high.
For the source element, we should have the `:output` pads format which would allow us to send video in the higher and in the lower quality. The same format should be specified on the input of the filter element. However, the stream format on the output of the filter should accept only video in the lower quality.
Here is the definition of the source element:

```elixir
# Source element

defmodule Source do
 def_output_pad(:output,
   demand_unit: :buffers,
   stream_format:  any_of([
        %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480, height: 300}
        when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60,
        %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 720, height: 480}
        when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60
      ])
 )
 ...
 def handle_playing(_context, state) do
 ...
   { {[stream_format: {:output, %Formats.Raw{pixel_format: I420, framerate: 45, width: 720, height: 300} }]}, state}
 end
```

While returning from the `handle_playing/2` callback, the element will send the format described by the `Formats.Raw` structure, through the `:output` pad.
Will this format meet the accepted specification provided by us? Think about it!
In fact, it will, as the `Formats.Raw` structure sent with `:stream_format` action matches the pattern - the value of `:pixel_format` field is one of `:I420` and `:I422`, and the `:framerate` is in the range between 30 and 60. In case the structure didn't match the pattern, a runtime exception would be thrown.

Below there is the draft of the filter implementation:

```elixir
# Filter

defmodule Filter do
 def_input_pad:input,
   demand_unit: :buffers,
   accepted_format: any_of([
          %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480, height: 300}
          when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60,
          %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 720, height: 480}
          when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60
        ])

  def_output_pad :output,
    demand_unit: :buffers,
    accepted_format: %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480,height: 300} when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60

 ...

 def handle_stream_format(_pad, _stream_format, _context, state) do
  ...
  { {[stream_format: {:output, %Formats.Raw{pixel_format: I420, framerate: 60, width: 480, height: 300} }]}, state}
 end

end
```

When we receive the spec on the input pad, we do not propagate it to our `:output` pad - instead, we send a different format, with reduced quality (values of the `width` and `height` fields might be lower).

We hope by now you have a better understanding of what stream formats are. This knowledge will be helpful in the following chapters.


Stream format

# Stream formats in action

Since we have already discussed what stream formats are and how to use them, let's make use of them in our project!
The first thing to do is to define modules responsible for describing the formats used in our pipeline.
We will put them in a separate directory - `lib/formats`. Let's start with the format describing the [packets](../glossary/glossary.md#packet):

**_`lib/formats/packet_format.ex`_**

```elixir
defmodule Basic.Formats.Packet do
 @moduledoc """
 A module describing the format of the packet.
 """
 defstruct type: :custom_packets
end
```

The definition of the module is not complicated, as you can see in the code snippet above - we are only defining a structure within that module, with a `:type` parameter, whose default value is `:custom_packets`.

In our [pipeline](../glossary/glossary.md#pipeline) we will also send another type of data - [frames](../glossary/glossary.md#frame). Let's define a format for them:

**_`lib/formats/frame_format.ex`_**

```elixir
defmodule Basic.Formats.Frame do
 @moduledoc """
 A module describing the format of the frame.
 """
 defstruct encoding: :utf8
end
```

Same as in the case of the previous format - we are defining a structure with a single field, called `:encoding`, and the default value of that field - `:utf8`.

That's it! Format modules are really simple, and using them is not that hard either, as you will see in the subsequent chapters - defining the accepted stream formats and sending stream format actions!

Before advancing you can test the `Source` [element](../glossary/glossary.md/#source), using the tests provided in `/test` directory.

```console
mix test test/elements/source_test.exs
```

In case of errors, you may go back to the [Source chapter](/basic_pipeline/03_Source.md) or take a look how [Source](https://github.com/membraneframework/membrane_basic_pipeline_tutorial/blob/template/end/lib/elements/Source.ex) and [formats](https://github.com/membraneframework/membrane_basic_pipeline_tutorial/tree/template/end/lib/formats) should look like.


Formats

# Ordering Buffer

In this chapter we will deal with the next [element](../glossary/glossary.md#element) in our [pipeline](../glossary/glossary.md#pipeline) - the [Ordering Buffer](../glossary/glossary.md#jitter-buffer--ordering-buffer).
As stated in the [chapter about the system architecture](02_SystemArchitecture.md), this element is responsible for ordering the incoming [packets](../glossary/glossary.md#packet), based on their sequence id.
Because Ordering Buffer is a filtering element, we need to specify both the input and the output [pads](../glossary/glossary.md#pad):

**_`lib/elements/ordering_buffer.ex`_**

```elixir
defmodule Basic.Elements.OrderingBuffer do
  use Membrane.Filter
  alias Basic.Formats.Packet

  def_input_pad :input,
    flow_control: :manual,
    demand_unit: :buffers,
    accepted_format: %Packet{type: :custom_packets}

  def_output_pad :output,
    flow_control: :manual,
    accepted_format: %Packet{type: :custom_packets}
 ...
end
```

Note the format specification definition there - we expect `Basic.Formats.Packet` of type `:custom_packets` to be sent on the input pad, and the same type of packets to be sent through the output pad.
In the next step let's specify how we want the state of our element to look like:

**_`lib/elements/ordering_buffer.ex`_**

```elixir
defmodule Basic.Elements.OrderingBuffer do
 ...
  @impl true
  def handle_init(_context, _options) do
    {[],
     %{
       ordered_packets: [],
       last_sent_seq_id: 0
     }}
  end
 ...
end
```

If you don't remember what is the purpose of the Ordering Buffer, please refer to the [2nd chapter](02_SystemArchitecture.md).
We will need to hold a list of ordered packets, as well as a sequence id of the packet, which most recently was sent through the output pad (we need to know if there are some packets missing between the last sent packet and the first packet in our ordered list).

Handling demands is quite straightforward:

**_`lib/elements/ordering_buffer.ex`_**

```elixir
defmodule Basic.Elements.OrderingBuffer do
 ...
  @impl true
  def handle_demand(:output, size, _unit, _context, state) do
    {[demand: {:input, size}], state}
  end
 ...
end
```

We simply send the `:demand` on the `:input` pad once we receive a demand on the `:output` pad. One packet on input corresponds to one packet on output so for each 1 unit of demand we send 1 unit of demand to the `:input` pad.

Now we can go to the main part of the Ordering Buffer implementation - the `handle_buffer/4` callback.
The purpose of this callback is to process the incoming buffer. It gets called once a new buffer is available and waiting to be processed.

**_`lib/elements/ordering_buffer.ex`_**

```elixir
defmodule Basic.Elements.OrderingBuffer do
 ...
  @impl true
  def handle_buffer(:input, buffer, _context, state) do
    packet = unzip_packet(buffer.payload)
    ordered_packets = [packet | state.ordered_packets] |> Enum.sort()
    state = %{state | ordered_packets: ordered_packets}
    [{last_seq_id, _} | _] = ordered_packets
   ...
  end

  defp unzip_packet(packet) do
    regex = ~r/^\[seq\:(?<seq_id>\d+)\](?<data>.*)$/
    %{"data" => data, "seq_id" => seq_id} = Regex.named_captures(regex, packet)
    {String.to_integer(seq_id), %Membrane.Buffer{payload: data} }
  end
 ...
end
```

First, we are taking advantage of the [Regex module](https://hexdocs.pm/elixir/1.13/Regex.html) available in the Elixir.
With the `Regex.named_captures` we can access the values of the fields defined within the regex description.
Do you remember what our packet looks like?

```
[seq:7][frameid:2][timestamp:3]data
```

Above you can see an exemplary packet. We need to fetch the value of the sequence id (in our case it is equal to 7) and get the rest of the packet.
Therefore we have defined the regex description as:

```elixir
~r/^\[seq\:(?<seq_id>\d+)\](?<data>.*)$/
```

> ### TIP - How to read the regex?
>
> - `~r/.../` stands for the `sigil_r/1` [sigil](https://elixir-lang.org/getting-started/sigils.html)
> - `^` describes the beginning of the input
> - `\[` stands for the opening square bracket ('\[') at the beginning is required to escape the char since the plain '\[' has a special meaning in the regex syntax
> - `seq` is a sequence of 's', 'e', 'q' characters (we need to adjust our regex description to match the header of the packet)
> - `\:` stands for the ':' character (we also need to escape that character since it is meaningful in the regex's syntax)
> - `(?<seq_id>\d+)` allows us to define a named capture - later one, once we use the `Regex.named_captures/2`, we will retrieve the map with 'seq_id' key and the corresponding value equal to the string described by the `\d+` 'partial' regex (which means - one or more occurrences of a decimal). Generally speaking, a named capture can be specified with the following structure: `(?<key>regex)` where `regex` is a regex description.
> - `\]` is the escaped closing square bracket character
> - `(?<data>.*)` is a named capture description that allows us to get the value of a `.*` regex (any character no or any number of times) under a `data` key.
> - `$` stands for the end of the input

The result of `Regex.named_captures/2` applied to that regex description and the exemplary packet should be following:

```elixir
{"seq_id"=>7, "data"=>"[frameid:2][timestamp:3]data"}
```

Once we unzip the header of the packet in the `handle_buffer/4` callback, we can put the incoming packet in the `ordered_packets` list and sort that list. Due to the fact, that elements of this list are tuples, whose first element is a sequence id (a value that is unique), the list will be sorted based on the sequence id.
We also get the sequence id of the first element in the updated `ordered_packets` list.

Here comes the rest of the `handle_buffer/4` definition:

**_`lib/elements/ordering_buffer.ex`_**

```elixir
defmodule Basic.Elements.OrderingBuffer do
 ...
  def handle_buffer(:input, buffer, _context, state) do
 ...
    if state.last_sent_seq_id + 1 == last_seq_id do
      {ready_packets_sequence, ordered_packets_left} =
        get_ready_packets_sequence(ordered_packets, [])

      {last_sent_seq_id, _} = List.last(ready_packets_sequence)

      state = %{
        state
        | ordered_packets: ordered_packets_left,
          last_sent_seq_id: last_sent_seq_id
      }

      ready_buffers = Enum.map(ready_packets_sequence, &elem(&1, 1))

      {[buffer: {:output, ready_buffers}], state}
    else
      {[redemand: :output], state}
    end
  end
 ...
end
```

We need to distinguish between two situations: the currently processed packet can have a sequence id which is subsequent to the sequence id of the last sent packet or there might be some packets not yet delivered to us, with sequence ids in between the last sent sequence id and the sequence id of a currently processed packet. In the second case, we should store the packet and wait for the next packets to arrive. We will accomplish that using [`redemands` mechanism](../glossary/glossary.md#redemands), which will be explained in detail in the next chapter.
However, in the first situation, we need to get the ready packet's sequence - that means, a consistent batch of packets from the `:ordered_packets`. This can be done in the following way:

**_`lib/elements/ordering_buffer.ex`_**

```elixir
defmodule Basic.Elements.OrderingBuffer do
 ...
  defp get_ready_packets_sequence([first_packet | ordered_rest], []) do
    get_ready_packets_sequence(ordered_rest, [first_packet])
  end

  defp get_ready_packets_sequence(
         [next_seq = {next_id, _} | ordered_rest],
         [{last_id, _} | _] = ready_sequence
       )
       when next_id == last_id + 1 do
    get_ready_packets_sequence(ordered_rest, [next_seq | ready_sequence])
  end

  defp get_ready_packets_sequence(ordered_packets, ready_sequence) do
    {Enum.reverse(ready_sequence), ordered_packets}
  end
 ...
end
```

Note the order of the definitions, since we are taking advantage of the pattern matching mechanism!
The algorithm implemented in the snippet above is really simple - we are recursively taking the next packet out of the `:ordered_packets` buffer until it becomes empty or there is a missing packet (`next_id == last_id + 1`) between the last taken packet and the next packet in the buffer.
Once we have a consistent batch of packets, we can update the state (both the`:ordered_packets` and the `:last_sent_seq_id` need to be updated) and output the ready packets by defining the `:buffer` action.

Test the `OrderingBuffer`:

```console
mix test test/elements/ordering_buffer_test.exs
```

Now the `OrderingBuffer` element is ready. Before we implement the next element let us introduce you to the concept of redemands.


Ordering Buffer

# Redemands

Redemanding is a very convenient mechanism that helps you, the Membrane Developer, stick to the DRY (Don't Repeat Yourself) rule.
Generally speaking, it can be used in two situations:

- in the [source elements](../glossary/glossary.md#source)
- in the [filter elements](../glossary/glossary.md#filter)

To comprehensively understand the concept behind redemanding, you need to be aware of the typical control flow which occurs in Membrane's [elements](../glossary/glossary.md#element) - which you could have seen in the elements we have already defined.

## In Source elements

In the [source elements](../glossary/glossary.md#source), there is a "side-channel", from which we can receive data. That "side-channel" can be, as in the exemplary [pipeline](../glossary/glossary.md#pipeline) we are working on, in form of a file, from which we are reading the data. In real-life scenarios it could be also, i.e. an [RTP](../glossary/glossary.md#rtp) stream received via the network. Since we have that "side-channel", there is no need to receive data via the input [pad](../glossary/glossary.md#pad) (that is why we don't have it in the source element, do we?).
The whole logic of fetching the data can be put inside the `handle_demand/5` callback - once we are asked to provide the [buffers](../glossary/glossary.md#buffer), the `handle_demand/5` callback gets called and we can provide the desired number of buffers from the "side-channel", inside the body of that callback. No processing occurs here - we get asked for the buffer and we provide the buffer, simple as that.
The redemand mechanism here lets you focus on providing a single buffer in the `handle_demand/5` body - later on, you can simply return the `:redemand` action and that action will invoke the `handle_demand/5` once again, with the updated number of buffers which are expected to be provided. Let's see it in an example - we could have such a `handle_demand/5` definition (and it wouldn't be a mistake!):

```elixir
@impl true
def handle_demand(:output, size, _unit, _context, state) do
 actions = for x <- 1..size do
   payload = Input.get_next() #Input.get_next() is an exemplary function which could be providing data
   {:buffer, %Membrane.Buffer(payload: payload)}
 end
 { {:ok, actions}, state}
end
```

As you can see in the snippet above, we need to generate the required `size` of buffers in the single `handle_demand/5` run. The logic of supplying the demand there is quite easy - but what if you would also need to check if there is enough data to provide a sufficient number of buffer? You would need to check it in advance (or try to read as much data as possible before supplying the desired number of buffers). And what if an exception occurs during the generation, before supplying all the buffers?
You would need to take under the consideration all these situations and your code would become larger and larger.

Wouldn't it be better to focus on a single buffer in each `handle_demand/5` call - and let the Membrane Framework automatically update the demand's size? This can be done in the following way:

```elixir
@impl true
def handle_demand(:output, _size, unit, context, state) do
 payload = Input.get_next() #Input.get_next() is an exemplary function which could be providing data
 actions = [buffer: %Membrane.Buffer(payload: payload), redemand: :output]
 { {:ok, actions}, state}
end
```

## In Filter elements
In the filter element, the situation is quite different.
Since the filter's responsibility is to process the data sent via the input pads and transmit it through the output pads, there is no 'side-channel' from which we could take data. That is why in normal circumstances you would transmit the buffer through the output pad in the `handle_buffer/4` callback (which means - once your element receives a buffer, you process it, and then you 'mark' it as ready to be output with the `:buffer` action). When it comes to the `handle_demand/5` action on the output pad, all you need to do is to demand the appropriate number of buffers on the element's input pad. 

That behavior is easy to specify when we exactly know how many input buffers correspond to the one output buffer (recall the situation in the [Depayloader](../glossary/glossary.md#payloader-and-depayloader) of our pipeline, where we *a priori* knew, that each output buffer ([frame](../glossary/glossary.md#frame)) consists of a given number of input buffers ([packets](../glossary/glossary.md#packet))). However it becomes impossible to define if the output buffer might be a combination of a discretionary set number of input buffers. At the same time, we have dealt with an unknown number of required buffers in the OrderingBuffer implementation, where we didn't know how many input buffers do we need to demand to fulfill the missing spaces between the packets ordered in the list. How did we manage to do it? 

We simply used the `:redemand` action! In case there was a missing space between the packets, we returned the `:redemand` action, which immediately called the `handle_demand/5` callback (implemented in a way to request for a buffer on the input pad). The fact, that that callback invocation was immediate, which means - the callback was called synchronously, right after returning from the `handle_buffer/4` callback, before processing any other message from the element's mailbox - might be crucial in some situations, since it guarantees that the demand will be done before handling any other event.
Recall the situation in the [Mixer](../glossary/glossary.md#mixer), where we were producing the output buffers right in the `handle_demand/5` callback. We needed to attempt to create the output buffer after:

- updating the buffers' list in `handle_buffer/4`
- updating the status of the [track](../glossary/glossary.md#track) in `handle_end_of_stream/3`
  Therefore, we were simply returning the `:redemand` action, and the `handle_demand/5` was called sequentially after on, trying to produce the output buffer.

As you can see, redemand mechanism in filters helps us deal with situations, where we do not know how many input buffers to demand in order to be able to produce an output buffer/buffers.
In case we don't provide enough buffers in the `handle_demand/5` callback (or we are not sure that we do provide), we should call `:redemand` somewhere else (usually in the `handle_buffer/4`) to make sure that the demand is not lost.

With that knowledge let's carry on with the next element in our pipeline - `Depayloader`.


Redemands

# Depayloader
Since we have [packets](../glossary/glossary.md#packet) put in order by the [Ordering Buffer](../basic_pipeline/06_OrderingBuffer.md), we can assemble them into the original [frames](../glossary/glossary.md#frame).
The Depayloader is an element responsible for this task. Specifically speaking, it unpacks the payload from the packets -
and that is why it's called 'depayloader'.
Let's create a new module in the `lib/elements/depayloader.ex` file:

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 use Membrane.Filter

 alias Basic.Formats.{Packet, Frame}
 ...
end
```

What input data do we expect? Of course in `Basic.Format.Packet` format!

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  def_input_pad :input,
    flow_control: :manual,
    demand_unit: :buffers,
    accepted_format: %Packet{type: :custom_packets}
 ...
end
```

However, our element will process that input data in a way that will change the format - on output, there will be frames instead of packets!
We need to specify it while defining the `:output` pad:

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  def_output_pad :output,
    flow_control: :manual,
    accepted_format: %Frame{encoding: :utf8}
 ...
end
```

We will also need a parameter describing how many packets should we request once we receive a demand for a frame:

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  def_options packets_per_frame: [
                spec: pos_integer,
                description:
                  "Positive integer, describing how many packets form a single frame. Used to demand the proper number of packets while assembling the frame."
              ]
 ...
end
```

In the `handle_init/2` callback we are simply saving the value of that parameter in the state of our element:

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  @impl true
  def handle_init(_context, options) do
    {[],
     %{
       frame: [],
       packets_per_frame: options.packets_per_frame
     }}
  end
 ...
end
```

Within the state, we will also hold a (potentially not complete) `:frame` - a list of packets, which form a particular frame. We will aggregate the packets in the `:frame` until the moment the frame is complete.

As noted in the [chapter dedicated to stream formats](04_StreamFormat.md), since we are changing the type of data within the element, we cannot rely on the default implementation of the `handle_stream_format/4` callback. We need to explicitly send the updated version of the format:

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  @impl true
  def handle_stream_format(:input, _stream_format, _context, state) do
    {[stream_format: {:output, %Frame{encoding: :utf8}}], state}
  end
 ...
end
```

As in most elements, the `handle_demand/5` implementation is quite easy - what we do is simply make a demand on our `:input` pad once we receive a demand on the `:output` pad. However, since we are expected to produce a frame (which is formed from a particular number of packets) on the `:output` pad, we need to request a particular number of packets on the `:input` pad - that is why we have defined the `:packets_per_frame` option and now we will be making use of it. In case we would have been asked to produce 10 frames, and each frame would have been made out of 5 packets, then we would need to ask for 10\*5 = 50 packets on the `:input`.

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  @impl true
  def handle_demand(:output, size, :buffers, _context, state) do
    {[demand: {:input, size * state.packets_per_frame}], state}
  end
 ...
end
```

There is nothing left apart from processing the input data - that is - the packets. Since the packets are coming in order, we can simply hold them in the `:frame` list until all the packets forming that frame will be there. As you might remember, each packet has a frame id in its header, which can be followed by a 'b' or 'e' character, indicating the type of the packet (the one beginning a frame or the one ending the frame). We will use information about the type to find a moment in which we should produce a frame out of the packets list.

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  @impl true
  def handle_buffer(:input, buffer, _context, state) do
    packet = buffer.payload

    regex =
      ~r/^\[frameid\:(?<frame_id>\d+(?<type>[s|e]*))\]\[timestamp\:(?<timestamp>\d+)\](?<data>.*)$/

    %{"data" => data, "frame_id" => _frame_id, "type" => type, "timestamp" => timestamp} =
      Regex.named_captures(regex, packet)

    frame = [data | state.frame]
 ...
end
```

Once again we are taking advantage of the `Regex.named_captures`.
Once we fetch the interesting values of the header's parameters, we can update the `:frame`.

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  @impl true
  def handle_buffer(:input, buffer, _context, state) do
   ...
    if type == "e" do
      buffer = %Membrane.Buffer{
        payload: prepare_frame(frame),
        pts: String.to_integer(timestamp)
      }

      {[buffer: {:output, buffer}], %{state | frame: []}}
    else
      {[], %{state | frame: frame}}
    end
 ...
end
```

Now, depending on the type of frame, we perform different actions.
If we have the 'ending' packet, we are making the `:buffer` action with the frame made out of the packets (that's where `prepare_frame/1` function comes in handy), and clear the `:frame` buffer. Here is how the `prepare_frame/1` function can be implemented:

**_`lib/elements/depayloader.ex`_**

```elixir
defmodule Basic.Elements.Depayloader do
 ...
  defp prepare_frame(frame) do
    frame |> Enum.reverse() |> Enum.join("")
  end
 ...
end
```

Otherwise, if the packet is not of the 'ending' type (that is - it can be both the 'beginning' frame or some packet in the middle), we are simply updating the state with the processed packet added to the `:frame` buffer and redemanding packet.

Test the `Depayloader`:

```console
mix test test/elements/depayloader_test.exs
```

With the [`Source`](../glossary/glossary.md#source), [`OrderingBuffer`](../glossary/glossary.md#jitter-buffer--ordering-buffer) and [`Depayloader`](../glossary/glossary.md#payloader-and-depayloader) elements ready we are able to read packets from file, order them based on their sequence ID and assemble them back into frames.
In the next chapter we will be dealing with the [`Mixer`](../glossary/glossary.md#mixer) which will merge two message streams in order to create complete conversation.


Depayloader

# Mixer

Here comes the mixer - an [element](../glossary/glossary.md#element) responsible for mixing two streams of [frames](../glossary/glossary.md#frame), coming from two different sources.
Once again we start with defining the initialization options and the pads of both types:

**_`lib/elements/mixer.ex`_**

```elixir
defmodule Basic.Elements.Mixer do
  @moduledoc """
  Element responsible for mixing the frames coming from two sources, basing on their timestamps.
  """
  use Membrane.Filter
  alias Basic.Formats.Frame

  def_input_pad :first_input,
    flow_control: :manual,
    demand_unit: :buffers,
    accepted_format: %Frame{encoding: :utf8}

  def_input_pad :second_input,
    flow_control: :manual,
    demand_unit: :buffers,
    accepted_format: %Frame{encoding: :utf8}

  def_output_pad :output,
    flow_control: :manual,
    accepted_format: %Frame{encoding: :utf8}
 ...
end
```

Note, that we have defined two input [pads](../glossary/glossary.md#pad): `:first_input` and the `:second_input`.
Each of these input pads will have a corresponding incoming [track](../glossary/glossary.md#track) in form of a [buffers](../glossary/glossary.md#buffer) stream. We need a structure that will hold the state of the track. Let's create it by defining a `Track` inside the mixer module:

**_`lib/elements/mixer.ex`_**

```elixir
defmodule Basic.Elements.Mixer do
 ...
  defmodule Track do
   @type t :: %__MODULE__{
     buffer: Membrane.Buffer.t(),
     status: :started | :finished
   }
   defstruct buffer: nil, status: :started
  end
 ...
end
```

As you can see in the code snippet above, the `Track` will consist of the `:buffer` field, holding the very last buffer received on the corresponding input pad, and the `:status` fields, indicating the status of the track - `:started`, in case we are still expecting some buffers to come (that means - in case `:end_of_stream` event hasn't been received yet) and `:finished` otherwise. 

It's a good practice to provide a type specification for such a custom struct since it makes the code easier to reuse and lets the compiler warn us about some misspellings (for instance in the status field atoms), which cause some hard to spot errors. 

A careful reader might notice, that we are holding only one buffer for each track, instead of a list of all the potentially unprocessed buffers - does it mean that we are losing some of them? Not at all, since we are taking advantage of the elements which have appeared earlier in the [pipeline](../glossary/glossary.md#pipeline) and which provide us with an ordered list of frames on each of the inputs - however, we will need to process each buffer just at the moment it comes on the pad.

The logic we're going to implement can be described in the following three steps:

- If all the tracks are in the 'active' state ('active' means - the ones in the `:started` state or the ones in the `:finished` state but with an unprocessed buffer in the `Track` structure) - output the one with the lower timestamp. Otherwise do nothing.
- If all the tracks are in the `:finished` state and their `:buffer` is empty - send the `:end_of_stream` event.
- For all the tracks which are in the `:started` state and their buffer is empty - demand on the pad corresponding to that track.

The next step in our element implementation is quite an obvious one:

**_`lib/elements/mixer.ex`_**

```elixir
defmodule Basic.Elements.Mixer do
 ...
  @impl true
  def handle_init(_context, _options) do
    {[],
     %{
       tracks: %{first_input: %Track{}, second_input: %Track{}}
     }}
  end
 ...
end
```

We have provided a `handle_init/2` callback, which does not expect any options to be passed. We are simply setting up the structure of the element state.
As mentioned previously, we will have a `Track` structure for each of the input pads. 

What's interesting is this is where the mixer having exactly two inputs stops being important. The missing functionality can be defined generically without much hassle. 

Following on the callbacks implementation, let's continue with the `handle_buffer/4` implementation:

**_`lib/elements/mixer.ex`_**

```elixir
defmodule Basic.Elements.Mixer do
 ...
  @impl true
  def handle_buffer(pad, buffer, _context, state) do
    tracks = Map.update!(state.tracks, pad, &%Track{&1 | buffer: buffer})
    {tracks, buffer_actions} = get_output_buffers_actions(tracks)
    state = %{state | tracks: tracks}

    {buffer_actions ++ [redemand: :output], state}
  end
 ...
end
```

In this callback we update the mixer's state by assigning the incoming buffer to its track. We can be sure no overwriting of an existing buffer happens, which will become more apparent as we delve further into the logic's implementation. 

Once the state is updated we gather all buffers that can be sent (might be none) in `get_output_buffers_actions/1` and return the coresponding `buffer` actions. In case any demands should be sent afterwards we also tell the output pad to redemand.

**_`lib/elements/mixer.ex`_**

```elixir
defmodule Basic.Elements.Mixer do
 ...
  @impl true
  def handle_end_of_stream(pad, _context, state) do
    tracks = Map.update!(state.tracks, pad, &%Track{&1 | status: :finished})
    {tracks, buffer_actions} = get_output_buffers_actions(tracks)
    state = %{state | tracks: tracks}

    if Enum.all?(tracks, fn {track_id, track} ->
         track.status == :finished and not has_buffer?({track_id, track})
       end) do
      {buffer_actions ++ [end_of_stream: :output], state}
    else
      {buffer_actions ++ [redemand: :output], state}
    end
  end
 ...
end
```

What we did here was similar to the logic defined in `handle_buffer/4` - we have just updated the state of the track (in that case - by setting its status to `:finished`), gather the buffers and send them. The important difference is that in case all inputs have closed, we should forward an `end_of_stream` action instead of a `redemand`, signaling the mixer has finished its processing. 

Let's now implement gathering ready buffers:

**_`lib/elements/mixer.ex`_**

```elixir
defmodule Basic.Elements.Mixer do
 ...
  defp has_buffer?({_track_id, track}),
    do: track.buffer != nil

  defp can_send_buffer?(tracks) do
    started_tracks =
      Enum.filter(
        tracks,
        fn {_track_id, track} -> track.status != :finished end
      )

    (started_tracks == [] and Enum.any?(tracks, &has_buffer?/1)) or
      (started_tracks != [] and Enum.all?(started_tracks, &has_buffer?/1))
  end

  defp get_output_buffers_actions(tracks) do
    {buffers, tracks} = prepare_buffers(tracks)
    buffer_actions = Enum.map(buffers, fn buffer -> {:buffer, {:output, buffer}} end)
    {tracks, buffer_actions}
  end

  defp prepare_buffers(tracks) do
    if can_send_buffer?(tracks) do
      {next_track_id, next_track} =
        tracks
        |> Enum.filter(&has_buffer?/1)
        |> Enum.min_by(fn {_track_id, track} -> track.buffer.pts end)

      tracks = Map.put(tracks, next_track_id, %Track{next_track | buffer: nil})
      {buffers, tracks} = prepare_buffers(tracks)
      {[next_track.buffer | buffers], tracks}
    else
      {[], tracks}
    end
  end
end
```

The `prepare_buffers/1` function is the most involved here, so let's start with that. We first check whether we can send a buffer at all. The next buffer to send in order will of course be one with lowest `.pts`. We then empty the corresponding track's buffer. There might be more than one buffer ready to send and so we iterate the gathering recursively. 

We define `can_send_buffer?` as follows. If there's any `:started` track still waiting on a buffer we cannot send more, since whatever buffers the mixer's currently holding might come after the one that's yet to be received on this track. 

Otherwise, if all tracks have finished it can still be the case that some have non-empty buffers. We can happily send all of these in order since it is guaranteed there is no buffer preceding them that we would have to wait for.

All that's left now is to handle redemands.

**_`lib/elements/mixer.ex`_**

```elixir
defmodule Basic.Elements.Mixer do
 ...
  def handle_demand(:output, _size, _unit, context, state) do
    demand_actions =
      state.tracks
      |> Enum.reject(&has_buffer?/1)
      |> Enum.filter(fn {track_id, track} ->
        track.status != :finished and context.pads[track_id].demand == 0
      end)
      |> Enum.map(fn {track_id, _track} -> {:demand, {track_id, 1}} end)

    {demand_actions, state}
  end
 ...
end
```

Since it should be responsible for producing and sending `demand` actions to the corresponding input pads, we accordingly filter tracks for ones that are empty, started, and with no demands pending. 
It should also become clearer why in `handle_buffer/4` the receiving track is sure to have an empty buffer ready to be overwritten, since we only send demands to input pads of empty tracks.

And that's all! the mixer's ready to mix, and ready to be tested:

```console
mix test test/elements/mixer_test.exs
```

Now all that's left to do is to save our stream to file using [`Sink`](../glossary/glossary.md#sink).


Mixer

# Sink

The sink is the last [element](../glossary/glossary.md#element) in our [pipeline](../glossary/glossary.md#pipeline), designed to store the data processed by the pipeline.
In contrast to the [filter elements](../glossary/glossary.md#filter), it won't have any output [pad](../glossary/glossary.md#pad) - that is why we need to make our element `use Membrane.Sink` and define the input pad only.
Since we want to parameterize the usage of that element, it will be good to define the options structure, so that we can specify the path to the file where the output should be saved. This stuff is done in the code snippet below:

**_`lib/elements/sink.ex`_**

```elixir
defmodule Basic.Elements.Sink do
  use Membrane.Sink

  def_options location: [
                spec: String.t(),
                description: "Path to the file"
              ]

  def_input_pad :input,
    flow_control: :manual,
    demand_unit: :buffers,
    accepted_format: _any
 ...
end
```

No surprises there - now we need to specify the element's behavior by defining the relevant callbacks!

**_`lib/elements/sink.ex`_**

```elixir

defmodule Basic.Elements.Sink do
 ...
  @impl true
  def handle_init(_context, options) do
    {[], %{location: options.location}}
  end
 ...
end
```

We have started with `handle_init/2`, where we are initializing the state of the element (we need to store the path to the output files).

Later on, we can specify the `handle_playing/2` callback - this callback gets called once the pipeline's setup finishes - that is a moment when we can demand the [buffers](../glossary/glossary.md#buffer) for the first time (since the pipeline is already prepared to work):

**_`lib/elements/sink.ex`_**

```elixir

defmodule Basic.Elements.Sink do
 ...
  @impl true
  def handle_playing(_context, state) do
    {[demand: {:input, 10}], state}
  end
 ...
end
```

There is only one more callback that needs to be specified - `handle_buffer/4`, which get's called once there are some buffers that can be processed (which means, that there are buffers to be written since there are no output pads through which we could be transmitting these buffers):

**_`lib/elements/sink.ex`_**

```elixir

defmodule Basic.Elements.Sink do
 ...
  @impl true
  def handle_buffer(:input, buffer, _context, state) do
    File.write!(state.location, buffer.payload <> "\n", [:append])
    {[demand: {:input, 10}], state}
  end
end
```

Note that after the successful writing, we are taking the `:demand` action and we ask for some more buffer.

With the `Sink` completed, we have implemented all elements of our pipeline. Now let's move to the very last step - defining the actual pipeline using the elements we have created.


Sink

# Pipeline

The time has come to assemble all the bricks together and create the pipeline!
This task is really easy since the Membrane Framework provides a sort of DSL (*Domain Specific Language*) which allows you to link the prefabricated components together.
In many real-life scenarios, this part would be the only thing you would need to do since you can take advantage of plenty of ready components (in form of [elements](../glossary/glossary.md#element) and [bins](../glossary/glossary.md#bin)) which are available as a part of the Membrane Framework. For now, we will create the pipeline out of the elements we have created during that tutorial!

## Defining the pipeline

The pipeline is another behavior introduced by the Membrane Framework. To make the module a pipeline, we need to make it `use Membrane.Pipeline`. That is how we will start our implementation of the pipeline module, in the `lib/pipeline.ex` file:

**_`lib/pipeline.ex`_**

```elixir

defmodule Basic.Pipeline do

  use Membrane.Pipeline
 ...
end
```

You could have guessed - all we need to do now is to describe the desired behavior by implementing the callbacks! In fact, the only callback we want to implement if the pipeline is the `handle_init/2` callback, called once the pipeline is initialized - of course, there are plenty of other callbacks which you might find useful while dealing with a more complex task. You can read about them [here](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#callbacks).
Please add the following callback signature to our `Basic.Pipeline` module:

**_`lib/pipeline.ex`_**

```elixir

defmodule Basic.Pipeline do
 ...
 @impl true
 def handle_init(_context, _options) do

 end
end
```

As you can see, we can initialize the pipeline with some options, but in our case, we do not need them.

## Supervising and children-parent relationship

In Elixir's actor model, derived from the Erlang programming language (as well as in many other implementations of the actor system) there is a concept of the actors supervising each other.
In case of the actor failing due to an error it is its supervisor's responsibility to deal with that fact - either by stopping that actor, restarting it, or performing some other action.
With such a concept in mind, it's possible to create reliable and fault-tolerant actor systems.
[Here](https://blog.appsignal.com/2021/08/23/using-supervisors-to-organize-your-elixir-application.html) there is a really nice article describing that concept and providing an example of the actor system. Feel free to stop here and read about the supervision mechanism in Elixir if you have never met with that concept before.
Our pipeline will also be an actor system - with the `Basic.Pipeline` module being the supervisor of all its elements.
As you have heard before - it is the supervisor's responsibility to launch its children's processes.
In the Membrane Framework's pipeline there is a special action designed for that purpose - [`:spec`](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html#t:spec/0).
As you can see, you need to specify the `Membrane.ChildrenSpec` for that purpose.

Please stop for a moment and read about the [`Membrane.ChildrenSpec`](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html).
We will wait for you and once you are ready, we will define our own children and links ;)

Let's start with defining what children we need inside the `handle_init/2` callback! If you have forgotten what structure we want to achieve please refer to the [2nd chapter](02_SystemArchitecture.md) and recall what elements we need inside of our pipeline.

**_`lib/pipeline.ex`_**

```elixir
defmodule Basic.Pipeline do
 ...
  @impl true
  def handle_init(_context, _options) do
    spec = [
      child(:input1, %Basic.Elements.Source{location: "input.A.txt"})
      |> child(:ordering_buffer1, Basic.Elements.OrderingBuffer)
      |> child(:depayloader1, %Basic.Elements.Depayloader{packets_per_frame: 4})
      |> via_in(:first_input)
      |> child(:mixer, Basic.Elements.Mixer)
      |> child(:output, %Basic.Elements.Sink{location: "output.txt"}),
      child(:input2, %Basic.Elements.Source{location: "input.B.txt"})
      |> child(:ordering_buffer2, Basic.Elements.OrderingBuffer)
      |> child(:depayloader2, %Basic.Elements.Depayloader{packets_per_frame: 4})
      |> via_in(:second_input)
      |> get_child(:mixer)
    ]

    {[spec: spec], %{}}
  end
 ...
```

Remember to pass the desired file paths in the `:location` option! 

Now... that's it! :) 
The spec list using Membrane's DSL is enough to describe our pipeline's topology. The child keywords spawn components of the specified type and we can use the `|>` operator to link them together. When the pads that should be linked are unamibiguous this is straightforward but for links like those with `Mixer` we can specify the pads using `via_in/1`. There also exists a `via_out/1` keyword which works similarly for output pads. 

As you can see the first argument to `child/2` is a component identifier. If we had omitted it Membrane would generate a unique identifier under the hood. For more about the `child` functions please refer to [the docs](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#child/1).

Our pipeline is ready! Let's try to launch it.
We will do so by starting the pipeline, and then playing it. For the ease of use we will do it in a script.

**_`start.exs`_**

```elixir
{:ok, _sup, pipeline} = Membrane.Pipeline.start_link(Basic.Pipeline)
Process.monitor(pipeline)

# Wait for the pipeline to terminate
receive do
  {:DOWN, _monitor, :process, ^pipeline, _reason} -> :ok
end
```

You can execute it by running `mix run start.exs` in the terminal.

In the output file (the one specified in the `handle_init/2` callback of the pipeline) you should see the recovered conversation.

In case of any problems you can refer to the code on the `template/end` branch of `membrane_basic_pipeline_tutorial` repository.

Now our solution is completed. You have acquired a basic understanding of Membrane, and you can implement a simple pipeline using elements.

If you wish to extend your knowledge of Membrane's concepts we encourage you to read the [extension to this tutorial](../basic_pipeline_extension/01_Introduction.md).


Pipeline

All you need to know about pipelines pt 1

Additional features you might find usefull while setting up your pipelines


In this section, we gathered some topics which should give you better insight into Membrane. We will base on the ["All you need to know about pipelines pt 1" tutorial](../basic_pipeline/01_Introduction.md), so make sure you have gone through it first.
The chapters are loosely related, so feel free to read them in any order you want.

**Table of contents**

- [Bin](../basic_pipeline_extension/02_Bin.md) - how to create a reusable group of [elements](../glossary/glossary.md#element)
- [Dynamic Pads](../basic_pipeline_extension/03_DynamicPads.md) - a more flexible way to define element's [pads](../glossary/glossary.md#pad)
- [Tests](../basic_pipeline_extension/04_Tests.md) - creating unit tests for the elements


Introductions

# Bin

A Membrane's bin is a container for [elements](../glossary/glossary.md#element), which allows for creating reusable groups of elements.
Bin is similar to a [pipeline](../glossary/glossary.md#pipeline) in that it consists of linked elements. Such bin can then be placed inside a pipeline and linked with other entities - elements or bins. Bins can also be nested within one another.
Bin also has another advantage - it manages its children, for instance by dynamically spawning or replacing them as the stream changes.

## Enclosing pipeline elements inside a bin

As you can see, we have [`Source`](../glossary/glossary.md#source) -> [`Ordering Buffer`](../glossary/glossary.md#jitter-buffer--ordering-buffer) -> [`Depayloader`](../glossary/glossary.md#payloader-and-depayloader) chain, which is duplicated.
![Pipeline scheme](assets/images/basic_pipeline.png) <br>

We can encapsulate these elements inside `Bin`.
![Pipeline scheme using bin](assets/images/basic_pipeline_bin.png) <br>

Notice that there is no direct connection between `Depayloader` and [`Mixer`](../glossary/glossary.md#mixer). We have to explicitly link the `Depayloader` with `Bin`'s output [pads](../glossary/glossary.md#pad) and then we will connect the output pads to `Mixer`'s input pads.

Let's define the bin's output pads and its elements.

**_`lib/Bin.ex`_**

```elixir
defmodule Basic.Bin do
  use Membrane.Bin

  def_output_pad :output, accepted_format: %Basic.Formats.Frame{encoding: :utf8}

  def_options input_filename: [
              type: :string,
              description: "Input file for conversation."
            ]

  @impl true
  def handle_init(_ctx, options) do
    spec = [
      child(:input, %Basic.Elements.Source{location: options.input_filename})
      |> child(:ordering_buffer, Basic.Elements.OrderingBuffer)
      |> child(:depayloader, %Basic.Elements.Depayloader{packets_per_frame: 4}) 
      |> bin_output(:output)
    ]

    {[spec: spec] %{} }
  end
end
```

The output pads of the bin are matching the ones we [defined for depayloader](/basic_pipeline/08_Depayloader.md#libelementsdepayloaderex-2).
Notice that the last link is between `depayloader` and the bin's output pads. In general, if we wanted to receive data in a bin we would need to link the first processing component in the bin with the `bin_input(<bin's input pad name>)`.

## Modifying pipeline using bin

Using the bin we created, we can replace the elements in the pipeline.

**_`lib/Pipeline.ex`_**


```elixir
defmodule Basic.Pipeline do
  @moduledoc """
  A module providing the pipeline, which aggregates and links the elements.
  """
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, _opts) do
    spec = [
      child(:bin1, %Basic.Bin{input_filename: "input.A.txt"}) 
      |> via_in(:first_input)
      |> child(:mixer, Basic.Elements.Mixer),
      child(:bin2, %Basic.Bin{input_filename: "input.B.txt"}) 
      |> via_in(:second_input) 
      |> get_child(:mixer),
      get_child(:mixer) 
      |> child(:output, %Basic.Elements.Sink{location: "output.txt"})
    ]

    {[spec: spec], %{}}
  end
end
```

Combining the usage of the bin and [dynamic pads](03_DynamicPads.md) will result in an even cleaner and more scalable solution.


# Dynamic Pads

The solution we have implemented along the tutorial has at least one downside - it is definitely not easily extendable.
What if we needed to support mixing streams coming from three different speakers in the conversation?
Well, we would need to add another input [pad](../glossary/glossary.md#pad) in the [Mixer](../glossary/glossary.md#mixer), for instance - `:third_input` pad, and then update our [pipeline](../glossary/glossary.md#pipeline) definition:

**_`lib/Pipeline.ex`_**

```elixir
@impl true
def handle_init(_ctx, _opts) do
 spec = [
   ...
   child(:input3, %Basic.Elements.Source{location: "input.C.txt"}) 
   |> child(:ordering_buffer3, Basic.Elements.OrderingBuffer) 
   |> child(:depayloader3, %Basic.Elements.Depayloader{packets_per_frame: 4}),
   ...
   get_child(:depayloader3) 
   |> via_in(:third_input) 
   |> get_child(:mixer),
 ]
 ...
end
```

But what if there were 4 people in the conversation? Adding another pad to the Mixer would solve the problem, but this solution does not scale well.
And what if the number of speakers was unknown at the moment of the compilation? Then we wouldn't be able to predefine the pads in the Mixer.
The Membrane Framework comes with a solution for this sort of problem - and the solution is called: **dynamic pads**.

## What is the idea behind the dynamic pads?

Well, the idea is quite simple! Instead of specifying a single pad with a predefined name (*static pad*) as we did in all the modules before, we specify, that we want **a set of pads** of a given type. Initially, that set will be empty, but with each link created in the parent's child specification, the [element](../glossary/glossary.md#element) will be informed, that the pad of a given type was added - and therefore it will be able to invoke the `handle_pad_added/3` callback.

## The Mixer revisited

Let's try to use dynamic pads for the input in the Mixer!
The very first thing we need to do is to use the `def_input_pads` appropriately.

**_`lib/elements/Mixer.ex`_**

```elixir
...
def_input_pad :input, 
   demand_unit: :buffers, 
   flow_control: :manual, 
   availability: :on_request, 
   accepted_format: %Basic.Formats.Frame{encoding: :utf8}
...
```

We have added the [`availability: :on_request` option](https://hexdocs.pm/membrane_core/Membrane.Pad.html#t:availability/0), which allows us to define the set of dynamic pads, identified as `:input`.

No more do we have the `:first_input` and the `:second_input` pads defined, so we do not have the [tracks](../glossary/glossary.md#track) corresponding to them either! Let's update the `handle_init/2` callback:

**_`lib/elements/Mixer.ex`_**

```elixir
...
@impl true
def handle_init(_ctx, _options) do
    {[], %{ tracks: %{} }}
end
...
```

Tracks map is initially empty since there are no corresponding pads.
The next thing we need to do is to implement the `handle_pad_added/3` callback, which will be called once the pipeline starts, with some links pointing to dynamic pads:

**_`lib/elements/Mixer.ex`_**

```elixir
...
@impl true
def handle_pad_added(pad, _context, state) do
 state = %{state | tracks: Map.put(state.tracks, pad, %Track{})}
 {[], state}
end
...
```

Once a pad is created, we add a new `Track` to the tracks map, with the pad being its key.
That's it! Since we have already designed the Mixer in a way it is capable of serving more tracks, there is nothing else to do.

## Updated pipeline

Below you can find the updated version of the pipeline's `handle_init/2` callback:

**_`lib/Pipeline.ex`_**

```elixir
...
@impl true
def handle_init(_ctx, _opts) do
 spec = [
    child(:input1, %Basic.Elements.Source{location: "input.A.txt"}) 
    |> child(:ordering_buffer1, Basic.Elements.OrderingBuffer) 
    |> child(:depayloader1, %Basic.Elements.Depayloader{packets_per_frame: 4}),
    child(:input2, %Basic.Elements.Source{location: "input.B.txt"}) 
    |> child(:ordering_buffer2, Basic.Elements.OrderingBuffer) 
    |> child(:depayloader2, %Basic.Elements.Depayloader{packets_per_frame: 4}),
    get_child(:depayloader1) 
    |> via_in(Pad.ref(:input, :first))
    |> child(:mixer, Basic.Elements.Mixer),
    get_child(:depayloader2) 
    |> via_in(Pad.ref(:input, :second)) 
    |> get_child(:mixer),
    get_child(:mixer) 
    |> child(:output, %Basic.Elements.Sink{location: "output.txt"})
 ]

 {[spec: spec], %{}}
end
...
```

The crucial thing was to change the plain atom name identifying the pad (like `:first_input`) into the [Membrane.Pad.ref/2](https://hexdocs.pm/membrane_core/Membrane.Pad.html#ref/2).
The first argument passed to that function is the name of the dynamic pad's set (in our case: `:input`, as we have defined the `:input` dynamic pads set in the Mixer element), and the second argument is a particular pad identifier.
As you can see, we have created two `:input` pads: `:first` and `:second`. While starting the pipeline, the `handle_pad_added/3` callback will be called twice, once per each dynamic pad created.

## Further actions

As an exercise, you can try to modify the `lib/pipeline.ex` file and define a pipeline consisting of three parallel branches, being mixed in a single Mixer. Later on, you can check if the pipeline works as expected, by generating the input files out of the conversation in which participate three speakers.

If you combine the approach taken in the chapter about [Bin](02_Bin.md) you can simplify this solution by reducing the size of the link defintions inside the pipeline.


Dynamic Pads

# Tests

When creating elements for the basic pipeline you used the provided tests.
In this chapter we will explain in detail how they work, and give you with some good practices which will allow you to write reliable tests for your Membrane system.

## Do we need testing?

Testing in terms of software engineering is no less important than functionalities programming.
We are sure we do not need to persuade you that our [pipeline](../glossary/glossary.md#pipeline), and especially its elements need testing. In fact - who wouldn't like to be sure, that changes just made in the functionalities code do not break the desired element's behavior? And all that in the matter of typing simple `mix test` command?

In the scope of this chapter we will implement some unit tests for the elements of our pipeline. They will check the behavior of the elements in isolation.
We will immediately jump to the code and try to experience the Membrane tests on our own.

## Our first test

Elixir comes with a great tool for testing - [ExUnit](https://hexdocs.pm/ex_unit/ExUnit.html).
If you have never written tests in ExUnit, feel free to stop for a moment and read about how the tests are constructed - however, ExUnit deals with tests in such a clear way that probably you will be able to see what's going on by just looking at the code snippets in the further part of this tutorial.
We need to specify, that we will be writing only unit tests - that means, we will write tests checking the behavior of a single [element](../glossary/glossary.md#element), isolated from the other elements in the pipeline.
Let's create a `test/elements/depayloader_test.exs` file and put the following code inside it:

**_`test/elements/depayloader_test.exs`_**

```elixir
defmodule DepayloaderTest do
 use ExUnit.Case 
 doctest Basic.Elements.Depayloader

 test "Depayloader should assemble the packets and form a frame" do
  ...
 end
end
```

This way we are defining a testing module with the use of the ExUnit, as well as a single test (we will put the test logic inside the inner `do...end` scope).
The `doctest` macro checks if the given module has proper documentation (typespecs, module description etc.).

We have decided to show the process of writing tests based on the depayloader - since the behavior of this element is well described and can be easily checked. Let's recall what is the responsibility of the depayloader - this element is receiving ordered [packets](../glossary/glossary.md#packet) and is about to form [frames](../glossary/glossary.md#frame) out of them. Let's check if it is doing it properly!

**_`test/elements/depayloader_test.exs`_**

```elixir
defmodule DepayloaderTest do
 ...
 alias Basic.Elements.Depayloader
 alias Membrane.Buffer
 
 test "Depayloader should assemble the packets and form a frame" do
  {[], state} = Depayloader.handle_init(nil, %Depayloader{packets_per_frame: 5})

  {[], state} =
    Depayloader.handle_buffer(
      :input,
      %Buffer{payload: "[frameid:1s][timestamp:1]Hello! "},
      nil,
      state
    )

  {[], state} =
    Depayloader.handle_buffer(
      :input,
      %Buffer{payload: "[frameid:1][timestamp:1]How are"},
      nil,
      state
    )

  {actions, _state} =
    Depayloader.handle_buffer(
      :input,
      %Buffer{payload: "[frameid:1e][timestamp:1] you?"},
      nil,
      state
    )

  [buffer: {:output, buffer}] = actions
  assert buffer.payload == "Hello! How are you?"
 end
end
```

We have been explicitly calling the callbacks defined in the `Depayloader` module, with the appropriate arguments.
First, we have initialized the state by calling the `handle_init/2` callback, later on, we have made the depayloader `handle_buffer/4` some [buffers](../glossary/glossary.md#buffer) (note that we had to explicitly pass the `%Membrane.Buffer{}` structure as the argument of the function invocation as well as keep track of the state, which gets updated after each call).
Finally, after the last buffer is processed (the last buffer is the buffer whose `frameid` should contain the `e` letter meaning that that is the packet that is ending a given frame), we are expecting the action to be returned - and this action should be a `:buffer` actions, transmitting the buffer with the complete frame through the `:output` pad. We also assert (using the ExUnit's [`assert`](https://hexdocs.pm/ex_unit/ExUnit.Assertions.html#assert/1) macro) the value hold in the buffer's payload (It should be a complete sentence).
If no action was returned from the last `handle_buffer/4`, the pattern wouldn't match to the `{ {actions, _state}`, and the test would fail.
If the assertion on the output buffer's payload wouldn't be true - the test would also fail.

## The Membrane's support for tests

The test written above is quite simple. Probably you have noticed that what we did there was "simulating" the behavior of the Membrane's Core, in a limited, but satisfying our needs, way. During the pipeline run, it is Membrane's responsibility to invoke the callbacks and pass the updated version of the state as the argument.
However, Membrane's Core behavior is much more complicated - if we were using some more complex mechanism (i.e. [redemands](../glossary/glossary.md#redemands)), possibly we would need to simulate that behavior in a more detailed way - finally ending with the Membrane's Core in our test module.
Such an approach scales terribly - and that is why we want to avoid it. Membrane Core's developers have given us support for testing the elements which allow us to have a simple pipeline consisting of a generic [source](../glossary/glossary.md#source) and [sink](../glossary/glossary.md#sink), as well as our element.
Such a pipeline behaves just like any other pipeline in a regular working Membrane system - however, we are also given a bunch of helpful tools (like assertion macros) to check if our element has a desired business logic implemented in its behavior.
Below we will rewrite the test we have just written, but with the support from the Membrane Framework:

**_`test/elements/depayloader_test.exs`_**

```elixir
defmodule DepayloaderTest do
  ...
  alias Membrane.Buffer

  alias Membrane.Testing.{Source, Sink}

  import Membrane.Testing.Assertions
  alias Membrane.Testing.{Source, Sink, Pipeline}
  alias Basic.Formats.Packet

  test "Depayloader should assemble the packets and form a frame (with membrane's testing framework)" do
    inputs = [
      "[frameid:1s][timestamp:1]Hello! ",
      "[frameid:1][timestamp:1]How are",
      "[frameid:1e][timestamp:1] you?"
    ]

    generator = fn state, size ->
      if state == [] do
        {[end_of_stream: :output], state}
      else
        [payload | new_state] = state

        if size > 1 do
          {[buffer: {:output, %Buffer{payload: payload}}, redemand: :output], new_state}
        else
          {[buffer: {:output, %Buffer{payload: payload}}], new_state}
        end
      end
    end

    spec =
      child(:source, %Source{
        output: {initial_state, generator},
        stream_format: %Packet{type: :custom_packets}
      })
      |> child(:depayloader, %Depayloader{packets_per_frame: 5})
      |> child(:sink, Sink)

    pipeline = Pipeline.start_link_supervised!(spec: spec)
    assert_start_of_stream(pipeline, :sink)

    assert_sink_buffer(pipeline, :sink, %Buffer{payload: "Hello! How are you?"})

    assert_end_of_stream(pipeline, :sink)
    refute_sink_buffer(pipeline, :sink, _, 2000)
    Pipeline.terminate(pipeline)
 end
end
```

First, we have defined a `:inputs` list, consisting of the messages which will be wrapped by the `Membrane.Buffer` and used to "feed" our element.
Later on, we have specified the testing pipeline.
Our testing pipeline consists only of three elements - the source, the sink, and the element we are about to test.
We are specifying these elements by passing the list of link as a `:spec` option to the `Membrane.Testing.Pipeline`, in a similar manner as we do when we return the list of links with the `:spec` action in a regular pipeline.
The generic [`Membrane.Testing.Source`](https://hexdocs.pm/membrane_core/Membrane.Testing.Source.html) accepts `:output` field as one of its options - we can pass the list of payloads which will be sent through the `:output` pad of the testing or a ("generator")[https://hexdocs.pm/membrane_core/Membrane.Testing.Source.html#t:generator/0]. In our case we use the "generator" which allows us to pass a function describing the source behaviour.
Once the pipeline structure is defined, we can start the pipeline process.
And here comes the assertions section - we are taking advantage of some [Membrane specific assertions](https://hexdocs.pm/membrane_core/Membrane.Testing.Assertions.html):

- first, we are asserting that the stream has started, with the `assert_start_of_stream/2`
- then we are asserting that the sink has received a buffer of a given form (in our case - we want the sink to receive the buffer with a frame assembled out of the input packets) - with help of `assert_sink_buffer/3`
- then we are asserting that `:end_of_stream` has reached the `:sink` - with `assert_end_of_stream/2`
- the last assertion we made is that the `:sink` hasn't received any buffer within 2000 milliseconds - and `refute_sink_buffer/4` helps us do it

Finally, we need to stop and terminate our pipeline. It is a good practice to do it in a blocking manner so that the test returns after the pipeline is terminated (the `Membrane.Pipeline.terminate` works in a blocking manner by default).

At the first glance, this might look like a little bit of overkill to use the Membrane's testing framework - the amount of code in this particular test has swollen enormously!
But that is just because the functionality we are testing is quite simple.
Keep in mind that in the second test we are making some additional, more complicated assertions - just imagine if you were about to check if no buffer has reached the `:sink` after the `:end_of_stream` was sent - it wouldn't be that easy, would it?
With the Membrane's testing framework you can do it in one line only!

Now we can run the tests with a simple [mix](../glossary/glossary.md#mix) task, by typing:

```console
mix test
```

If everything works (both the tests and the functionality's code itself), you should see a notification that the test has passed successfully, which we hope you do see!

## Some special types of tests

As you remember, Source and Sink elements act specifically different than the [Filter elements](../glossary/glossary.md#filter) - that is why they are communicating with the 'outer world', i.e. by reading the data from a file or saving the result to the file. In order to check if their behavior is desired, we cannot create a testing pipeline with generic Source and Sink, since it is a Source/Sink that we want to test.
We will need to somehow mock the `outer environment` - let's see how this can be done, based on the example of the Source test:

**_`test/elements/source_test.exs`_**

```elixir
defmodule SourceTest do
 use ExUnit.Case, async: false
  import Mock
  alias Basic.Elements.Source
  alias Membrane.Buffer

  doctest Basic.Elements.Source

  @exemplary_content ["First Line", "Second Line"]
  @exemplary_location "path/to/file"
  @options %Source{location: @exemplary_location}
 ...

  test "reads the input file correctly" do
    with_mock File, read!: fn _ -> "First Line\nSecond Line" end do
      {_actions, state} =
        Source.handle_setup(nil, %{location: @exemplary_location, content: nil})

      assert state.content == @exemplary_content
    end
 end
```

We take advantage of the [Mock](https://hexdocs.pm/mock/Mock.html) library which is designed to help us substitute the function invocations.
As you can see in the code snippet above, we have mocked the invocations of the `File.read!` function - inside the scope of `with_mock/2` they are always returning `"First Line\nSecondLine"`.
That makes our test so much easier since we do not need to create a mock file with some content meant just for testing - the whole test is defined inside one file.

Would you be able to write some more tests for the other pipeline's elements yourself? Or perhaps you could try to extend the tests for the element we have just checked in action?
Give it a try, as it is a great exercise that will not only examine if you are familiar with testing the Membrane's system but also - do you understand functionalities of the elements and what is the element generic behavior!

When writing your tests for other elements feel free to take inspiration from the ones we provided in the `test/` directory. You can find them [here](https://github.com/membraneframework/membrane_basic_pipeline_tutorial/tree/template/end/test).


Tests

All you need to know about pipelines pt 2

In this short guide we provide you with an overview of how to create your own Membrane plugin and how to integrate it into your project.

During the development of Membrane Framework we aim at designing fewer, but higher quality plugins. However, we also kept extendability and reusability in mind. That's why it is easy for developers like you to create their own custom plugin, which satisfies their needs.

In this short guide we provide you with an overview of how to create your own Membrane plugin and how to integrate it into your project.

## Membrane plugin template

To create a new plugin, we recommend using the [template](https://github.com/membraneframework/membrane_template_plugin) that has been made for this very purpose and which will be the base of your plugin.
It defines necessary dependencies as well as other project specs, e.g. formatting, and guarantees you compliance with other Membrane components.

You can start creating a plugin by making your copy of the template. Go to the [github repo](https://github.com/membraneframework/membrane_template_plugin) and select `Use this template`. Then choose an appropriate name for the project.

If you haven't already, we suggest you read [basic pipeline tutorial](/basic_pipeline/create_new_plugin.md) to get familiar with Membrane's plugin structure. In any case, as you might have guessed the code of your plugin will go into `/lib` directory and the tests belong in the `/test` directory.

## Utilizing your plugin in a project

When your plugin is ready for being integrated into another project you can simply add it as a dependency in `mix.exs` as described [here](https://hexdocs.pm/mix/Mix.Tasks.Deps.html). Here's what it can look like:

```elixir
defp deps do
    [
      ...
      {:your_membrane_plugin, git: "https://github.com/githubuser/your_membrane_plugin", tag: "0.1"} # dependency from github
      {:your_membrane_plugin, ">=0.1.0"} # dependency from [hex](https://hex.pm/)
      {:your_membrane_plugin, path: "path/to/your_plugin"} # dependency from local file
      ...
    ]
end
```

And just like this, you have added your plugin to a project.


Create New Plugin

How to create your own plugin

All you need to know to understand digital video. Written by Leonardo Moreira.

This tutorial is essential for anyone who wants not only create multimedia applications, but also who wants to understand concepts of digital video processing and streaming
<br/>
Admittedly, it's not us who created this tutorial but we found it containing all the aspects we wanted to write about, so we're glad we can share it with you here.
<br/>
All credits to [Leaonardo Moreira](https://github.com/leandromoreira) and his contributors. You can find original files of this tutorial here: [Digital Video Introduction on Github](https://github.com/leandromoreira/digital_video_introduction)
<br/>


Preface

# Intro

A gentle introduction to video technology, although it's aimed at software developers / engineers, we want to make it easy **for anyone to learn**. This idea was born during a [mini workshop for newcomers to video technology](https://docs.google.com/presentation/d/17Z31kEkl_NGJ0M66reqr9_uTG6tI5EDDVXpdPKVuIrs/edit#slide=id.p).

The goal is to introduce some digital video concepts with a **simple vocabulary, lots of visual elements and practical examples** when possible, and make this knowledge available everywhere. Please, feel free to send corrections, suggestions and improve it.

There will be **hands-on** sections which require you to have **docker installed** and this repository cloned.

```bash
git clone https://github.com/leandromoreira/digital_video_introduction.git
cd digital_video_introduction
./setup.sh
```
> **WARNING**: when you see a `./s/ffmpeg` or `./s/mediainfo` command, it means we're running a **containerized version** of that program, which already includes all the needed requirements.

All the **hands-on should be performed from the folder you cloned** this repository. For the **jupyter examples** you must start the server `./s/start_jupyter.sh` and copy the URL and use it in your browser.



Intro

# Basic terminology

An **image** can be thought of as a **2D matrix**. If we think about **colors**, we can extrapolate this idea seeing this image as a **3D matrix** where the **additional dimensions** are used to provide **color data**.

If we chose to represent these colors using the [primary colors (red, green and blue)](https://en.wikipedia.org/wiki/Primary_color), we define three planes: the first one for **red**, the second for **green**, and the last one for the **blue** color.

![an image is a 3d matrix RGB](/i/image_3d_matrix_rgb.png "An image is a 3D matrix")

We'll call each point in this matrix **a pixel** (picture element). One pixel represents the **intensity** (usually a numeric value) of a given color. For example, a **red pixel** means 0 of green, 0 of blue and maximum of red. The **pink color pixel** can be formed with a combination of the three colors. Using a representative numeric range from 0 to 255, the pink pixel is defined by **Red=255, Green=192 and Blue=203**.

> #### Other ways to encode a color image
> Many other possible models may be used to represent the colors that make up an image. We could, for instance, use an indexed palette where we'd only need a single byte to represent each pixel instead of the 3 needed when using the RGB model. In such a model we could use a 2D matrix instead of a 3D matrix to represent our color, this would save on memory but yield fewer color options.
>
> ![NES palette](/i/nes-color-palette.png "NES palette")

For instance, look at the picture down below. The first face is fully colored. The others are the red, green, and blue planes (shown as gray tones).

![RGB channels intensity](/i/rgb_channels_intensity.png "RGB channels intensity")

We can see that the **red color** will be the one that **contributes more** (the brightest parts in the second face) to the final color while the **blue color** contribution can be mostly **only seen in Mario's eyes** (last face) and part of his clothes, see how **all planes contribute less** (darkest parts) to the **Mario's mustache**.

And each color intensity requires a certain amount of bits, this quantity is known as **bit depth**. Let's say we spend **8 bits** (accepting values from 0 to 255) per color (plane), therefore we have a **color depth** of **24 bits** (8 bits * 3 planes R/G/B), and we can also infer that we could use 2 to the power of 24 different colors.

> **It's great** to learn [how an image is captured from the world to the bits](http://www.cambridgeincolour.com/tutorials/camera-sensors.htm).

Another property of an image is the **resolution**, which is the number of pixels in one dimension. It is often presented as width × height, for example, the **4×4** image below.

![image resolution](/i/resolution.png "image resolution")

> #### Hands-on: play around with image and color
> You can [play around with image and colors](/image_as_3d_array.ipynb) using [jupyter](#how-to-use-jupyter) (python, numpy, matplotlib and etc).
>
> You can also learn [how image filters (edge detection, sharpen, blur...) work](/filters_are_easy.ipynb).

Another property we can see while working with images or video is the **aspect ratio** which simply describes the proportional relationship between width and height of an image or pixel.

When people says this movie or picture is **16x9** they usually are referring to the **Display Aspect Ratio (DAR)**, however we also can have different shapes of individual pixels, we call this **Pixel Aspect Ratio (PAR)**.

![display aspect ratio](/i/DAR.png "display aspect ratio")

![pixel aspect ratio](/i/PAR.png "pixel aspect ratio")

> #### DVD is DAR 4:3
> Although the real resolution of a DVD is 704x480 it still keeps a 4:3 aspect ratio because it has a PAR of 10:11 (704x10/480x11)

Finally, we can define a **video** as a **succession of *n* frames** in **time** which can be seen as another dimension, *n* is the frame rate or frames per second (FPS).

![video](/i/video.png "video")

The number of bits per second needed to show a video is its **bit rate**.

> bit rate = width * height * bit depth * frames per second

For example, a video with 30 frames per second, 24 bits per pixel, resolution of 480x240 will need **82,944,000 bits per second** or 82.944 Mbps (30x480x240x24) if we don't employ any kind of compression.

When the **bit rate** is nearly constant it's called constant bit rate (**CBR**) but it also can vary then called variable bit rate (**VBR**).

> This graph shows a constrained VBR which doesn't spend too many bits while the frame is black.
>
> ![constrained vbr](/i/vbr.png "constrained vbr")

In the early days, engineers came up with a technique for doubling the perceived frame rate of a video display **without consuming extra bandwidth**. This technique is known as **interlaced video**; it basically sends half of the screen in 1 "frame" and the other half in the next "frame".

Today screens render mostly using **progressive scan technique**. Progressive is a way of displaying, storing, or transmitting moving images in which all the lines of each frame are drawn in sequence.

![interlaced vs progressive](/i/interlaced_vs_progressive.png "interlaced vs progressive")

Now we have an idea about how an **image** is represented digitally, how its **colors** are arranged, how many **bits per second** do we spend to show a video, if it's constant (CBR)  or variable (VBR), with a given **resolution** using a given **frame rate** and many other terms such as interlaced, PAR and others.

> #### Hands-on: Check video properties
> You can [check most of the  explained properties with ffmpeg or mediainfo.](https://github.com/leandromoreira/introduction_video_technology/blob/master/encoding_pratical_examples.md#inspect-stream)



Basic terminology

# Redundancy removal

We learned that it's not feasible to use video without any compression; **a single one hour video** at 720p resolution with 30fps would **require 278GB<sup>*</sup>**. Since **using solely lossless data compression algorithms** like DEFLATE (used in PKZIP, Gzip, and PNG), **won't** decrease the required bandwidth sufficiently we need to find other ways to compress the video.

> <sup>*</sup> We found this number by multiplying 1280 x 720 x 24 x 30 x 3600 (width, height, bits per pixel, fps and time in seconds)

In order to do this, we can **exploit how our vision works**. We're better at distinguishing brightness than colors, the **repetitions in time**, a video contains a lot of images with few changes, and the **repetitions within the image**, each frame also contains many areas using the same or similar color.

## Colors, Luminance and our eyes

Our eyes are [more sensitive to brightness than colors](http://vanseodesign.com/web-design/color-luminance/), you can test it for yourself, look at this picture.

![luminance vs color](/i/luminance_vs_color.png "luminance vs color")

If you are unable to see that the colors of the **squares A and B are identical** on the left side, that's fine, it's our brain playing tricks on us to **pay more attention to light and dark than color**. There is a connector, with the same color, on the right side so we (our brain) can easily spot that in fact, they're the same color.

> **Simplistic explanation of how our eyes work**
> The [eye is a complex organ](http://www.biologymad.com/nervoussystem/eyenotes.htm), it is composed of many parts but we are mostly interested in the cones and rods cells. The eye [contains about 120 million rod cells and 6 million cone cells](https://en.wikipedia.org/wiki/Photoreceptor_cell).
>
> To **oversimplify**, let's try to put colors and brightness in the eye's parts function. The **[rod cells](https://en.wikipedia.org/wiki/Rod_cell) are mostly responsible for brightness** while the **[cone cells](https://en.wikipedia.org/wiki/Cone_cell) are responsible for color**, there are three types of cones, each with different pigment, namely: [S-cones (Blue), M-cones (Green) and L-cones (Red)](https://upload.wikimedia.org/wikipedia/commons/1/1e/Cones_SMJ2_E.svg).
>
> Since we have many more rod cells (brightness) than cone cells (color), one can infer that we are more capable of distinguishing dark and light than colors.
>
> ![eyes composition](/i/eyes.jpg "eyes composition")
>
> **Contrast sensitivity functions**
>
> Researchers of experimental psychology and many other fields have developed many theories on human vision. And one of them is called Contrast sensitivity functions. They are related to spatio and temporal of the light and their value presents at given init light, how much change is required before an observer reported there was a change. Notice the plural of the word "function", this is for the reason that we can measure Contrast sensitivity functions with not only black-white but also colors. The result of these experiments shows that in most cases our eyes are more sensitive to brightness than color.

Once we know that we're more sensitive to **luma** (the brightness in an image) we can try to exploit it.

### Color model

We first learned [how to color images](#basic-terminology) work using the **RGB model**, but there are other models too. In fact, there is a model that separates luma (brightness) from  chrominance (colors) and it is known as **YCbCr**<sup>*</sup>.

> <sup>*</sup> there are more models which do the same separation.

This color model uses **Y** to represent the brightness and two color channels **Cb** (chroma blue) and **Cr** (chroma red). The [YCbCr](https://en.wikipedia.org/wiki/YCbCr) can be derived from RGB and it also can be converted back to RGB. Using this model we can create full colored images as we can see down below.

![ycbcr example](/i/ycbcr.png "ycbcr example")

### Converting between YCbCr and RGB

Some may argue, how can we produce all the **colors without using the green**?

To answer this question, we'll walk through a conversion from RGB to YCbCr. We'll use the coefficients from the **[standard BT.601](https://en.wikipedia.org/wiki/Rec._601)** that was recommended by the **[group ITU-R<sup>*</sup>](https://en.wikipedia.org/wiki/ITU-R)** . The first step is to **calculate the luma**, we'll use the constants suggested by ITU and replace the RGB values.

```
Y = 0.299R + 0.587G + 0.114B
```

Once we had the luma, we can **split the colors** (chroma blue and red):

```
Cb = 0.564(B - Y)
Cr = 0.713(R - Y)
```

And we can also **convert it back** and even get the **green by using YCbCr**.

```
R = Y + 1.402Cr
B = Y + 1.772Cb
G = Y - 0.344Cb - 0.714Cr
```

> <sup>*</sup> groups and standards are common in digital video, they usually define what are the standards, for instance, [what is 4K? what frame rate should we use? resolution? color model?](https://en.wikipedia.org/wiki/Rec._2020)

Generally, **displays** (monitors, TVs, screens and etc) utilize **only the RGB model**, organized in different manners, see some of them magnified below:

![pixel geometry](/i/new_pixel_geometry.jpg "pixel geometry")

### Chroma subsampling

With the image represented as luma and chroma components, we can take advantage of the human visual system's greater sensitivity for luma resolution rather than chroma to selectively remove information. **Chroma subsampling** is the technique of encoding images using **less resolution for chroma than for luma**.


![ycbcr subsampling resolutions](/i/ycbcr_subsampling_resolution.png "ycbcr subsampling resolutions")


How much should we reduce the chroma resolution?! It turns out that there are already some schemas that describe how to handle resolution and the merge (`final color = Y + Cb + Cr`).

These schemas are known as subsampling systems and are expressed as a 3 part ratio - `a:x:y` which defines the chroma resolution in relation to a `a x 2` block of luma pixels.

 * `a` is the horizontal sampling reference (usually 4)
 * `x` is the number of chroma samples in the first row of `a` pixels (horizontal resolution in relation to `a`)
 * `y` is the number of changes of chroma samples between the first and seconds rows of `a` pixels.

> An exception to this exists with 4:1:0, which provides a single chroma sample within each `4 x 4` block of luma resolution.

Common schemes used in modern codecs are: **4:4:4** *(no subsampling)*, **4:2:2, 4:1:1, 4:2:0, 4:1:0 and 3:1:1**.

> You can follow some discussions [to learn more about Chroma Subsampling](https://github.com/leandromoreira/digital_video_introduction/issues?q=YCbCr).

> **YCbCr 4:2:0 merge**
>
> Here's a merged piece of an image using YCbCr 4:2:0, notice that we only spend 12 bits per pixel.
>
> ![YCbCr 4:2:0 merge](/i/ycbcr_420_merge.png "YCbCr 4:2:0 merge")

You can see the same image encoded by the main chroma subsampling types, images in the first row are the final YCbCr while the last row of images shows the chroma resolution. It's indeed a great win for such small loss.

![chroma subsampling examples](/i/chroma_subsampling_examples.jpg "chroma subsampling examples")

Previously we had calculated that we needed [278GB of storage to keep a video file with one hour at 720p resolution and 30fps](#redundancy-removal). If we use **YCbCr 4:2:0** we can cut **this size in half (139 GB)**<sup>*</sup> but it is still far from ideal.

> <sup>*</sup> we found this value by multiplying width, height, bits per pixel and fps. Previously we needed 24 bits, now we only need 12.

<br/>

> ### Hands-on: Check YCbCr histogram
> You can [check the YCbCr histogram with ffmpeg.](/encoding_pratical_examples.md#generates-yuv-histogram) This scene has a higher blue contribution, which is showed by the [histogram](https://en.wikipedia.org/wiki/Histogram).
>
> ![ycbcr color histogram](/i/yuv_histogram.png "ycbcr color histogram")

### Color, luma, luminance, gamma video review

Watch this incredible video explaining what is luma and learn about luminance, gamma, and color.
[![Analog Luma - A history and explanation of video](http://img.youtube.com/vi/Ymt47wXUDEU/0.jpg)](http://www.youtube.com/watch?v=Ymt47wXUDEU)

> ### Hands-on: Check YCbCr intensity
> You can visualize the Y intensity for a given line of a video using [FFmpeg's oscilloscope filter](https://ffmpeg.org/ffmpeg-filters.html#oscilloscope).
> ```bash
> ffplay -f lavfi -i 'testsrc2=size=1280x720:rate=30000/1001,format=yuv420p' -vf oscilloscope=x=0.5:y=200/720:s=1:c=1
> ```
> ![y color oscilloscope](/i/ffmpeg_oscilloscope.png "y color oscilloscope")

## Frame types

Now we can move on and try to eliminate the **redundancy in time** but before that let's establish some basic terminology. Suppose we have a movie with 30fps, here are its first 4 frames.

![ball 1](/i/smw_background_ball_1.png "ball 1") ![ball 2](/i/smw_background_ball_2.png "ball 2") ![ball 3](/i/smw_background_ball_3.png "ball 3")
![ball 4](/i/smw_background_ball_4.png "ball 4")

We can see **lots of repetitions** within frames like **the blue background**, it doesn't change from frame 0 to frame 3. To tackle this problem, we can **abstractly categorize** them as three types of frames.

### I Frame (intra, keyframe)

An I-frame (reference, keyframe, intra) is a **self-contained frame**. It doesn't rely on anything to be rendered, an I-frame looks similar to a static photo. The first frame is usually an I-frame but we'll see I-frames inserted regularly among other types of frames.

![ball 1](/i/smw_background_ball_1.png "ball 1")

### P Frame (predicted)

A P-frame takes advantage of the fact that almost always the current picture can be **rendered using the previous frame.** For instance, in the second frame, the only change was the ball that moved forward. We can **rebuild frame 1, only using the difference and referencing to the previous frame**.

![ball 1](/i/smw_background_ball_1.png "ball 1") <-  ![ball 2](/i/smw_background_ball_2_diff.png "ball 2")

> #### Hands-on: A video with a single I-frame
> Since a P-frame uses less data why can't we encode an entire [video with a single I-frame and all the rest being P-frames?](/encoding_pratical_examples.md#1-i-frame-and-the-rest-p-frames)
>
> After you encoded this video, start to watch it and do a **seek for an advanced** part of the video, you'll notice **it takes some time** to really move to that part. That's because a **P-frame needs a reference frame** (I-frame for instance) to be rendered.
>
> Another quick test you can do is to encode a video using a single I-Frame and then [encode it inserting an I-frame each 2s](/encoding_pratical_examples.md#1-i-frames-per-second-vs-05-i-frames-per-second) and **check the size of each rendition**.

### B Frame (bi-predictive)

What about referencing the past and future frames to provide even a better compression?! That's basically what a B-frame is.

![ball 1](/i/smw_background_ball_1.png "ball 1") <-  ![ball 2](/i/smw_background_ball_2_diff.png "ball 2") -> ![ball 3](/i/smw_background_ball_3.png "ball 3")

> #### Hands-on: Compare videos with B-frame
> You can generate two renditions, first with B-frames and other with [no B-frames at all](/encoding_pratical_examples.md#no-b-frames-at-all) and check the size of the file as well as the quality.

### Summary

These frames types are used to **provide better compression**. We'll look how this happens in the next section, but for now we can think of **I-frame as expensive while P-frame is cheaper but the cheapest is the B-frame.**

![frame types example](/i/frame_types.png "frame types example")

## Temporal redundancy (inter prediction)

Let's explore the options we have to reduce the **repetitions in time**, this type of redundancy can be solved with techniques of **inter prediction**.


We will try to **spend fewer bits** to encode the sequence of frames 0 and 1.

![original frames](/i/original_frames.png "original frames")

One thing we can do it's a subtraction, we simply **subtract frame 1 from frame 0** and we get just what we need to **encode the residual**.

![delta frames](/i/difference_frames.png "delta frames")

But what if I tell you that there is a **better method** which uses even fewer bits?! First, let's treat the `frame 0` as a collection of well-defined partitions and then we'll try to match the blocks from `frame 0` on `frame 1`. We can think of it as **motion estimation**.

> ### Wikipedia - block motion compensation
> "**Block motion compensation** divides up the current frame into non-overlapping blocks, and the motion compensation vector **tells where those blocks come from** (a common misconception is that the previous frame is divided up into non-overlapping blocks, and the motion compensation vectors tell where those blocks move to). The source blocks typically overlap in the source frame. Some video compression algorithms assemble the current frame out of pieces of several different previously-transmitted frames."

![delta frames](/i/original_frames_motion_estimation.png "delta frames")

We could estimate that the ball moved from `x=0, y=25` to `x=6, y=26`, the **x** and **y** values are the **motion vectors**. One **further step** we can do to save bits is to **encode only the motion vector difference** between the last block position and the predicted, so the final motion vector would be `x=6 (6-0), y=1 (26-25)`

> In a real-world situation, this **ball would be sliced into n partitions** but the process is the same.

The objects on the frame **move in a 3D way**, the ball can become smaller when it moves to the background. It's normal that **we won't find the perfect match** to the block we tried to find a match. Here's a superposed view of our estimation vs the real picture.

![motion estimation](/i/motion_estimation.png "motion estimation")

But we can see that when we apply **motion estimation** the **data to encode is smaller** than using simply delta frame techniques.

![motion estimation vs delta ](/i/comparison_delta_vs_motion_estimation.png "motion estimation delta")

> ### How real motion compensation would look
> This technique is applied to all blocks, very often a ball would be partitioned in more than one block.
>  ![real world motion compensation](/i/real_world_motion_compensation.png "real world motion compensation")
> Source: https://web.stanford.edu/class/ee398a/handouts/lectures/EE398a_MotionEstimation_2012.pdf

You can [play around with these concepts using jupyter](/frame_difference_vs_motion_estimation_plus_residual.ipynb).

> #### Hands-on: See the motion vectors
> We can [generate a video with the inter prediction (motion vectors)  with ffmpeg.](/encoding_pratical_examples.md#generate-debug-video)
>
> ![inter prediction (motion vectors) with ffmpeg](/i/motion_vectors_ffmpeg.png "inter prediction (motion vectors) with ffmpeg")
>
> Or we can use the [Intel Video Pro Analyzer](https://software.intel.com/en-us/intel-video-pro-analyzer) (which is paid but there is a free trial version which limits you to only work with the first 10 frames).
>
> ![inter prediction intel video pro analyzer](/i/inter_prediction_intel_video_pro_analyzer.png "inter prediction intel video pro analyzer")

## Spatial redundancy (intra prediction)

If we analyze **each frame** in a video we'll see that there are also **many areas that are correlated**.

![](/i/repetitions_in_space.png)

Let's walk through an example. This scene is mostly composed of blue and white colors.

![](/i/smw_bg.png)

This is an `I-frame` and we **can't use previous frames** to predict from but we still can compress it. We will encode the red block selection. If we **look at its neighbors**, we can **estimate** that there is a **trend of colors around it**.

![](/i/smw_bg_block.png)

We will **predict** that the frame will continue to **spread the colors vertically**, it means that the colors of the **unknown pixels will hold the values of its neighbors**.

![](/i/smw_bg_prediction.png)

Our **prediction can be wrong**, for that reason we need to apply this technique (**intra prediction**) and then **subtract the real values** which gives us the residual block, resulting in a much more compressible matrix compared to the original.

![](/i/smw_residual.png)

There are many different types of this sort of prediction. The one you see pictured here is a form of straight planar prediction, where the pixels from the row above the block are copied row to row within the block. Planar prediction also can involve an angular component, where pixels from both the left and the top are used to help predict the current block. And there is also DC prediction, which involves taking the average of the samples right above and to the left of the block. 

> #### Hands-on: Check intra predictions
> You can [generate a video with macro blocks and their predictions with ffmpeg.](/encoding_pratical_examples.md#generate-debug-video) Please check the ffmpeg documentation to understand the [meaning of each block color](https://trac.ffmpeg.org/wiki/Debug/MacroblocksAndMotionVectors?version=7#AnalyzingMacroblockTypes).
>
> ![intra prediction (macro blocks) with ffmpeg](/i/macro_blocks_ffmpeg.png "inter prediction (motion vectors) with ffmpeg")
>
> Or we can use the [Intel Video Pro Analyzer](https://software.intel.com/en-us/intel-video-pro-analyzer) (which is paid but there is a free trial version which limits you to only work with the first 10 frames).
>
> ![intra prediction intel video pro analyzer](/i/intra_prediction_intel_video_pro_analyzer.png "intra prediction intel video pro analyzer")



Redundancy removal

How does a video codec work?

# Online streaming
## General architecture

![general architecture](/i/general_architecture.png "general architecture")

[TODO]

## Progressive download and adaptive streaming

![progressive download](/i/progressive_download.png "progressive download")

![adaptive streaming](/i/adaptive_streaming.png "adaptive streaming")

[TODO]

## Content protection

We can use **a simple token system** to protect the content. The user without a token tries to request a video and the CDN forbids her or him while a user with a valid token can play the content, it works pretty similarly to most of the web authentication systems.

![token protection](/i/token_protection.png "token_protection")

The sole use of this token system still allows a user to download a video and distribute it. Then the **DRM (digital rights management)** systems can be used to try to avoid this.

![drm](/i/drm.png "drm")

In real life production systems, people often use both techniques to provide authorization and authentication.

### DRM
#### Main systems

* FPS - [**FairPlay Streaming**](https://developer.apple.com/streaming/fps/)
* PR - [**PlayReady**](https://www.microsoft.com/playready/)
* WV - [**Widevine**](http://www.widevine.com/)


#### What?

DRM means [Digital rights management](https://sander.saares.eu/categories/drm-is-not-a-black-box/), it's a way **to provide copyright protection for digital media**, for instance, digital video and audio. Although it's used in many places [it's not universally accepted](https://en.wikipedia.org/wiki/Digital_rights_management#DRM-free_works).

#### Why?

Content creator (mostly studios) want to protect its intelectual property against copy to prevent unauthorized redistribution of digital media.

#### How?

We're going to describe an abstract and generic form of DRM in a very simplified way.

Given a **content C1** (i.e. an hls or dash video streaming), with a **player P1** (i.e. shaka-clappr, exo-player or ios) in a **device D1** (i.e. a smartphone, TV, tablet or desktop/notebook) using a **DRM system DRM1** (widevine, playready or FairPlay).

The content C1 is encrypted with a **symmetric-key K1** from the system DRM1, generating the **encrypted content C'1**.

![drm general flow](/i/drm_general_flow.jpeg "drm general flow")

The player P1, of a device D1, has two keys (asymmetric), a **private key PRK1** (this key is protected<sup>1</sup> and only known by **D1**) and a **public key PUK1**.

> **<sup>1</sup>protected**: this protection can be **via hardware**, for instance, this key can be stored inside a special (read-only) chip that works like [a black-box](https://en.wikipedia.org/wiki/Black_box) to provide decryption, or **by software** (less safe), the DRM system provides means to know which type of protection a given device has.


When the **player P1 wants to play** the **content C'1**, it needs to deal with the **DRM system DRM1**, giving its public key **PUK1**. The DRM system DRM1 returns the **key K1 encrypted** with the client''s public key **PUK1**. In theory, this response is something that **only D1 is capable of decrypting**.

`K1P1D1 = enc(K1, PUK1)`

**P1** uses its DRM local system (it could be a [SOC](https://en.wikipedia.org/wiki/System_on_a_chip), a specialized hardware or software), this system is **able to decrypt** the content using its private key PRK1, it can decrypt **the symmetric-key K1 from the K1P1D1** and **play C'1**. At best case, the keys are not exposed through RAM.

 ```
 K1 = dec(K1P1D1, PRK1)

 P1.play(dec(C'1, K1))
 ```

![drm decoder flow](/i/drm_decoder_flow.jpeg "drm decoder flow")



Online streaming

# How to use jupyter

Make sure you have **docker installed** and just run `./s/start_jupyter.sh` and follow the instructions on the terminal.



How to use jupyter

# Conferences

* [DEMUXED](https://demuxed.com/) - you can [check the last 2 events presentations.](https://www.youtube.com/channel/UCIc_DkRxo9UgUSTvWVNCmpA).



Conferences

# References

The richest content is here, it's where all the info we saw in this text was extracted, based or inspired by. You can deepen your knowledge with these amazing links, books, videos and etc.

Online Courses and Tutorials:

* https://www.coursera.org/learn/digital/
* https://people.xiph.org/~tterribe/pubs/lca2012/auckland/intro_to_video1.pdf
* https://xiph.org/video/vid1.shtml
* https://xiph.org/video/vid2.shtml
* https://wiki.multimedia.cx
* https://mahanstreamer.net
* http://slhck.info/ffmpeg-encoding-course
* http://www.cambridgeincolour.com/tutorials/camera-sensors.htm
* http://www.slideshare.net/vcodex/a-short-history-of-video-coding
* http://www.slideshare.net/vcodex/introduction-to-video-compression-13394338
* https://developer.android.com/guide/topics/media/media-formats.html
* http://www.slideshare.net/MadhawaKasun/audio-compression-23398426
* http://inst.eecs.berkeley.edu/~ee290t/sp04/lectures/02-Motion_Compensation_girod.pdf

Books:

* https://www.amazon.com/Understanding-Compression-Data-Modern-Developers/dp/1491961538/ref=sr_1_1?s=books&ie=UTF8&qid=1486395327&sr=1-1
* https://www.amazon.com/H-264-Advanced-Video-Compression-Standard/dp/0470516925
* https://www.amazon.com/High-Efficiency-Video-Coding-HEVC/dp/3319068946
* https://www.amazon.com/Practical-Guide-Video-Audio-Compression/dp/0240806301/ref=sr_1_3?s=books&ie=UTF8&qid=1486396914&sr=1-3&keywords=A+PRACTICAL+GUIDE+TO+VIDEO+AUDIO
* https://www.amazon.com/Video-Encoding-Numbers-Eliminate-Guesswork/dp/0998453005/ref=sr_1_1?s=books&ie=UTF8&qid=1486396940&sr=1-1&keywords=jan+ozer

Onboarding material:

* https://github.com/Eyevinn/streaming-onboarding
* https://howvideo.works/
* https://www.aws.training/Details/eLearning?id=17775
* https://www.aws.training/Details/eLearning?id=17887
* https://www.aws.training/Details/Video?id=24750

Bitstream Specifications:

* http://www.itu.int/rec/T-REC-H.264-201610-I
* http://www.itu.int/ITU-T/recommendations/rec.aspx?rec=12904&lang=en
* https://storage.googleapis.com/downloads.webmproject.org/docs/vp9/vp9-bitstream-specification-v0.6-20160331-draft.pdf
* http://iphome.hhi.de/wiegand/assets/pdfs/2012_12_IEEE-HEVC-Overview.pdf
* http://phenix.int-evry.fr/jct/doc_end_user/current_document.php?id=7243
* http://gentlelogic.blogspot.com.br/2011/11/exploring-h264-part-2-h264-bitstream.html
* https://forum.doom9.org/showthread.php?t=167081
* https://forum.doom9.org/showthread.php?t=168947

Software:

* https://ffmpeg.org/
* https://ffmpeg.org/ffmpeg-all.html
* https://ffmpeg.org/ffprobe.html
* https://mediaarea.net/en/MediaInfo
* https://www.jongbel.com/
* https://trac.ffmpeg.org/wiki/
* https://software.intel.com/en-us/intel-video-pro-analyzer
* https://medium.com/@mbebenita/av1-bitstream-analyzer-d25f1c27072b#.d5a89oxz8

Non-ITU Codecs:

* https://aomedia.googlesource.com/
* https://github.com/webmproject/libvpx/tree/master/vp9
* https://ghostarchive.org/archive/0W0d8 (was: https://people.xiph.org/~xiphmont/demo/daala/demo1.shtml)
* https://people.xiph.org/~jm/daala/revisiting/
* https://www.youtube.com/watch?v=lzPaldsmJbk
* https://fosdem.org/2017/schedule/event/om_av1/
* https://jmvalin.ca/papers/AV1_tools.pdf

Encoding Concepts:

* http://x265.org/hevc-h265/
* http://slhck.info/video/2017/03/01/rate-control.html
* http://slhck.info/video/2017/02/24/vbr-settings.html
* http://slhck.info/video/2017/02/24/crf-guide.html
* https://arxiv.org/pdf/1702.00817v1.pdf
* https://trac.ffmpeg.org/wiki/Debug/MacroblocksAndMotionVectors
* http://web.ece.ucdavis.edu/cerl/ReliableJPEG/Cung/jpeg.html
* http://www.adobe.com/devnet/adobe-media-server/articles/h264_encoding.html
* https://prezi.com/8m7thtvl4ywr/mp3-and-aac-explained/
* https://blogs.gnome.org/rbultje/2016/12/13/overview-of-the-vp9-video-codec/
* https://videoblerg.wordpress.com/2017/11/10/ffmpeg-and-how-to-use-it-wrong/

Video Sequences for Testing:

* http://bbb3d.renderfarming.net/download.html
* https://www.its.bldrdoc.gov/vqeg/video-datasets-and-organizations.aspx

Miscellaneous:

* https://github.com/Eyevinn/streaming-onboarding
* http://stackoverflow.com/a/24890903
* http://stackoverflow.com/questions/38094302/how-to-understand-header-of-h264
* http://techblog.netflix.com/2016/08/a-large-scale-comparison-of-x264-x265.html
* http://vanseodesign.com/web-design/color-luminance/
* http://www.biologymad.com/nervoussystem/eyenotes.htm
* http://www.compression.ru/video/codec_comparison/h264_2012/mpeg4_avc_h264_video_codecs_comparison.pdf
* https://web.archive.org/web/20100728070421/http://www.csc.villanova.edu/~rschumey/csc4800/dct.html (was: http://www.csc.villanova.edu/~rschumey/csc4800/dct.html)
* http://www.explainthatstuff.com/digitalcameras.html
* http://www.hkvstar.com
* http://www.hometheatersound.com/
* http://www.lighterra.com/papers/videoencodingh264/
* http://www.red.com/learn/red-101/video-chroma-subsampling
* http://www.slideshare.net/ManoharKuse/hevc-intra-coding
* http://www.slideshare.net/mwalendo/h264vs-hevc
* http://www.slideshare.net/rvarun7777/final-seminar-46117193
* http://www.springer.com/cda/content/document/cda_downloaddocument/9783642147029-c1.pdf
* http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/A-Progress-Report-The-Alliance-for-Open-Media-and-the-AV1-Codec-110383.aspx
* http://www.streamingmediaglobal.com/Articles/ReadArticle.aspx?ArticleID=116505&PageNum=1
* http://yumichan.net/video-processing/video-compression/introduction-to-h264-nal-unit/
* https://cardinalpeak.com/blog/the-h-264-sequence-parameter-set/
* https://cardinalpeak.com/blog/worlds-smallest-h-264-encoder/
* https://codesequoia.wordpress.com/category/video/
* https://developer.apple.com/library/content/technotes/tn2224/_index.html
* https://en.wikibooks.org/wiki/MeGUI/x264_Settings
* https://en.wikipedia.org/wiki/Adaptive_bitrate_streaming
* https://en.wikipedia.org/wiki/AOMedia_Video_1
* https://en.wikipedia.org/wiki/Chroma_subsampling#/media/File:Colorcomp.jpg
* https://en.wikipedia.org/wiki/Cone_cell
* https://en.wikipedia.org/wiki/File:H.264_block_diagram_with_quality_score.jpg
* https://en.wikipedia.org/wiki/Inter_frame
* https://en.wikipedia.org/wiki/Intra-frame_coding
* https://en.wikipedia.org/wiki/Photoreceptor_cell
* https://en.wikipedia.org/wiki/Pixel_aspect_ratio
* https://en.wikipedia.org/wiki/Presentation_timestamp
* https://en.wikipedia.org/wiki/Rod_cell
* https://it.wikipedia.org/wiki/File:Pixel_geometry_01_Pengo.jpg
* https://leandromoreira.com.br/2016/10/09/how-to-measure-video-quality-perception/
* https://sites.google.com/site/linuxencoding/x264-ffmpeg-mapping
* https://softwaredevelopmentperestroika.wordpress.com/2014/02/11/image-processing-with-python-numpy-scipy-image-convolution/
* https://tools.ietf.org/html/draft-fuldseth-netvc-thor-03
* https://www.encoding.com/android/
* https://www.encoding.com/http-live-streaming-hls/
* https://web.archive.org/web/20150129171151/https://www.iem.thm.de/telekom-labor/zinke/mk/mpeg2beg/whatisit.htm
* https://www.lifewire.com/cmos-image-sensor-493271
* https://www.linkedin.com/pulse/brief-history-video-codecs-yoav-nativ
* https://www.linkedin.com/pulse/video-streaming-methodology-reema-majumdar
* https://www.vcodex.com/h264avc-intra-precition/
* https://www.youtube.com/watch?v=9vgtJJ2wwMA
* https://www.youtube.com/watch?v=LFXN9PiOGtY
* https://www.youtube.com/watch?v=Lto-ajuqW3w&list=PLzH6n4zXuckpKAj1_88VS-8Z6yn9zX_P6
* https://www.youtube.com/watch?v=LWxu4rkZBLw
* https://web.stanford.edu/class/ee398a/handouts/lectures/EE398a_MotionEstimation_2012.pdf
* https://sander.saares.eu/categories/drm-is-not-a-black-box/


References

Digital Video Introduction

Set of terms we're using in our tutorials you might want to learn more about.

Glossary

Create your own media broadcasting solution!

We are glad you have decided to continue your journey into the oceans of multimedia with us!
In this tutorial, you will be able to see the Membrane in action - as we will prepare a pipeline responsible for converting
the incoming RTMP stream into an HLS stream, ready to be easily served on the internet.
Later on, we will substitute the front part of our pipeline, so that to make it compatible with another popular streaming protocol - RTSP.
That is how we will try to prove the great dose of flexibility that comes with the use of the Membrane Framework!

## General tutorial's structure
As mentioned earlier, the tutorial consists of two parts:
* "RTMP to HLS" - this part describes how to create a pipeline capable of receiving RTMP stream, muxing it into CMAF files, and serving them with the use of HTTP
* "RTSP to HLS" - this part is based on the previous one and describes how to change the pipeline from the previous part so that to make it capable of handling the RTSP stream instead of the RTMP stream.

As a result of each part, you will have a web application capable of playing media streamed with the use of the appropriate protocols - RTMP and RTSP.

We strongly encourage you to follow the tutorial part by part, as the different chapters are tightly coupled. However, in case you wanted just to see the result solutions or even one of them, we invite you to take a look at the appropriate directories of the `membrane_demo` repository:
* [RTMP to HLS](https://github.com/membraneframework/membrane_demo/tree/master/rtmp_to_hls)
* [RTSP to HLS](https://github.com/membraneframework/membrane_demo/tree/master/rtsp_to_hls)

General introduction

## What do we want to achieve?

Imagine that we need to build a video streaming platform. That means - we want a particular system user, the streamer, to stream its multimedia on a server, where it will be available for other participants - the viewers.
We need to take advantage of the fact, that there will be only one person streaming, and take into consideration, that there might be multiple, possibly many, viewers. What's the best solution for such a use case?
Well, as the tutorial name suggests - we can use RTMP for the streamer to stream its multimedia to the server, and make them accessible for the viewers by broadcasting them via HLS protocol!
Although it might sound quite complicated, with the Membrane Framework this will be easier than you can expect!
As a final product of this tutorial, we want to have a completely working solution for broadcasting a stream. It will consist of two parts:

- The pipeline, responsible for receiving an RTMP stream and preparing an HLS stream out of it
- The web player, capable of playing the HLS stream

The Membrane Framework will be advantageous in the first part where we will create a Membrane's Pipeline for converting the stream.
When it comes to the web player, we will use an existing solution - the [HLS.js](https://github.com/video-dev/hls.js/) player.

## Why one would need such a solution?

You might wonder why one would need to convert an RTMP stream into an HLS stream at all - couldn't we simply make the streamer broadcast its multimedia with the RTMP to all the viewers?
Technically speaking we could...but surprisingly it wouldn't be the easiest solution, since each of the viewers would need to act as an RTMP server. And definitely, it wouldn't be a solution that would scale - RTMP is based on TCP which implies, that there is no broadcast mechanism. It would be the streamer who would need to keep the direct connection to each of the viewers.
In contrast, the solution described above has plenty of advantages - the streamer needs to create a single connection with the RTMP server, and then the multimedia can be shared with the use of a regular HTTP server which is designed to serve multiple clients.

## A brief description of the technology we will use

As stated previously, we will use the preexisting protocols. You might find reading about them beneficial, and that is why we provide you with some links to a brief description of the technology we will be using:

- [How does RTMP work?](https://blog.stackpath.com/rtmp/) - if you are interested in how the connection in RTMP is established, take your time and read this short description!
- [What is RTMP and why should we care about it?](https://www.wowza.com/blog/rtmp-streaming-real-time-messaging-protocol) - here you can find another short description of RTMP, with the focus laid on the history of the protocol and comparison with other available protocols. Whatsmore, there is a comprehensive explanation of why we need to transcode RTMP to some HTTP-based protocol, just like HLS - which is the use case in our tutorial.
- [HLS behind the scenes](https://www.toptal.com/apple/introduction-to-http-live-streaming-hls) - dig into the ideas which stand behind the HLS, one of the most common HTTP-based streaming protocols.
- [Have you heard about CMAF?](https://www.wowza.com/blog/what-is-cmaf) - if you haven't make sure to read what is the purpose of having a `Common Media Application Format`!


RTMP introduction

As noted previously, our system will be divided into two parts:

- the server, which is responsible for receiving the RTMP stream, converting it into HLS, and then publishing the created files as an HTTP server
- the client, responsible for playing the incoming HLS stream.

Considering our point of view, the first part will be more complex, since we will need to prepare the processing pipeline on our own. Let's start with that part.

## The server

Below you can find the desired flow of data in our application server:
![Pipeline scheme](assets/RTMP_to_HLS_pipeline.drawio.png)

As you can see, the server architecture consists of two bigger processing units - in Membrane Framework we refer to such units as 'bins'.
The following bins will be used in our solution:

- RTMP Source Bin - responsible for receiving RTMP stream
- HLS Sink Bin - responsible for "packing" the data into the container format

Each of these bins consists of some subunits (in Membrane we refer to them as _elements_), responsible for completing an atomic multimedia processing task - i.e. parsing or payloading the incoming stream.

Let's take a quick walk through the whole processing line and describe more specifically what its given parts are meant to do.

### RTMP Source

The very first element, just at the beginning of the RTMP Source Bin is the RTMP Source. It acts as an RTMP server, listening for the incoming stream. It will be exposing a TCP port, and the streamer will be able to connect and start sending RTMP packets through that channel.
The incoming packets will be demuxed - meaning, that the packets will be unpacked and split, based on the track (video or audio) which data they are transporting.

## Parsers

Buffers containing the given track data will be sent to the appropriate parser - H264 parser for video data and AAC parser for audio data.

### H264 Parser

H264 parser is quite a complex element, designed to read the incoming H264 stream. We have prepared a [separate, supplemental chapter of this tutorial](13_H264_codec.md) to describe the H264 codec and our parser's implementation - we invite you to read it, however, that knowledge is not necessary to successfully run the application.

### AAC Parser

At the same time, parsing happens to the buffers containing audio data. Since the audio track has been encoded with the use of AAC, we need [AAC parser](https://github.com/membraneframework/membrane_aac_plugin) to decode it.
That part of the tutorial does not describe the AAC codec itself, and in case you are interested in digging into that codec, we highly recommend visiting [that page](https://wiki.multimedia.cx/index.php/Understanding_AAC).

### HLS converter

Once data reaches the HLS bin, it needs to be put into the appropriate container files. Since we will be using Common Media Application Format (CMAF) to distribute our media, we need to put all the tracks into the `fragmented MP4` container. The first step is to payload the track's data. Payloading transforms the media encoded with a given codec into a form that is suitable to be put into the container.
The audio track is payloaded within the `Membrane.MP4.Payloader.AAC` module and the video track is payloaded with an H264 payloader, implemented by the `Membrane.MP4.Payloader.H264` module.
The payloaded streams (both the audio stream and the video stream) are then put in the result CMAF file - and the `Membrane.MP4.Muxer.CMAF` is the element responsible for that process. The name 'muxer', relates to the process
of 'muxing' - putting multiple tracks in one file, which is done by that element. Furthermore, the `CMAF muxer`, splits the streams into so-called segments - stream parts of the desired duration. Along that procedure, the manifest files, which contain metadata about the media and names of the appropriate segment file names, are generated.
Finally, the output files: '.m4s' files for each of the segments, as well as manifest files, are written on the disk.
Let's take a look at how do the generated files look like:
![Pipeline scheme](assets/output_files_structure.png)
As we can see, there are:

- index.m3u8 - the manifest file which contains metadata about the media, as well as the URI of the manifest files for both the video and audio track.

```
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:TYPE=AUDIO,NAME="audio_default_name",GROUP-ID="audio_default_id",AUTOSELECT=YES,DEFAULT=YES,URI="audio.m3u8"
#EXT-X-STREAM-INF:BANDWIDTH=4507734,CODECS="avc1.42e00a",AUDIO="audio_default_id"
video_g2QABXZpZGVv.m3u8
```

- audio.m3u8 - the manifest file for the audio track.

```
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:8
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-DISCONTINUITY-SEQUENCE:0
#EXT-X-MAP:URI="audio_header_g2QABWF1ZGlv_part0_.mp4"
#EXTINF:8.0,
audio_segment_9_g2QABWF1ZGlv.m4s
#EXTINF:8.0,
audio_segment_10_g2QABWF1ZGlv.m4s
#EXTINF:8.0,
audio_segment_11_g2QABWF1ZGlv.m4s
```

That file contains some metadata (like the desired duration of the segment), along with the URI pointing to the '.mp4' header file of the fMP4 container, and the list of the segment files, in '.m4s' format.
Each segment is described with `EXTINF` directive, indicating the duration of the segment, in seconds.

- video\_<identifier>.m3u8 - the manifest file for the video track. Its structure is similar to the structure of the audio.m3u8 file. However, it is worth noting, that the desired duration of each segment is equal to 10 seconds - and the '.m4s' files are holding a video stream of that length.

```
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-DISCONTINUITY-SEQUENCE:0
#EXT-X-MAP:URI="video_header_g2QABXZpZGVv_part0_.mp4"
#EXTINF:10.0,
video_segment_8_g2QABXZpZGVv.m4s
#EXTINF:10.0,
video_segment_9_g2QABXZpZGVv.m4s
```

- audio_header\_<identifier>_part\<part_ordering_number>_.mp4 - a binary file, the header meant to describe the audio track in the format required by the fragmented MP4 container.
- video_header\_<identifier>\_part\<part_ordering_number>.mp4 - a binary file, the header meant to describe the video track in the format required by the fragmented MP4 container.
- audio_segment\_\<segment_ordering_number>\_\<>.m4s - particular segments containing fragments of the audio stream.
- video_segment\_\<segment_ordering_number>\_\<>.m4s - particular segments containing fragments of the video stream.

### HTTP server

The HTTP server sends the requested files to the clients. Its implementation is based on the API provided by the Phoenix Framework.

## The client

When it comes to our application client, we will be using the `Hls.js` library, available in the javascript ecosystem.
We won't dig into the details of how the client application media player is implemented, however, we would like to point out some steps which take place while playing the media.
First, the client needs to request the master manifest file. Based on that file, the client asks for the appropriate audio and video track manifest files.
With these files, the client knows the MP4 header files for both the tracks, as well as knows the filenames of the particular segments, along with their duration. The client downloads the MP4 header file and starts downloading the appropriate segment files, based on which part of the media should be played in the nearest future.


RTMP to HLS system architecture

This time we won't create the application from the scratch nor will we be based on some kind of template.
Since most of the application's code is media-agnostic, we don't see a point in focusing on it. Instead, we will indicate the most crucial parts of the
[RTMP to HLS demo](https://github.com/membraneframework/membrane_demo/tree/master/rtmp_to_hls), one of many publicly available [Membrane demos](https://github.com/membraneframework/membrane_demo/).
The Membrane demos are supplementing the Membrane Tutorials on providing a source of knowledge in the field of multimedia processing - we warmly invite you to look at them as you may find some interesting
use cases of Membrane in real-life scenarios.

## Running the application

In order to run the demo, you need to clone the Membrane demos repository first:

```console
git clone https://github.com/membraneframework/membrane_demo
cd membrane_demo/rtmp_to_hls
```

Once in the project directory, you need to get the dependencies of the project:

```console
mix deps.get
```

Finally, you can run the application with the following command:

```console
mix phx.server
```

The server will be waiting for an RTMP stream on localhost:9009, and the client of the application will be available on localhost:4000.

## Exemplary stream generation with OBS

You can send RTMP stream onto `localhost:9006` with your favorite streaming tool. As an example, we will show you how to generate an RTMP stream with
[OBS](https://obsproject.com).
Once you have OBS installed, you can perform the following steps:

1. Open the OBS application
1. Open the `Settings` windows
1. Go to the `Stream` tab and set the value in the `Server` field to: `rtmp://localhost:9006` (the address where the server is waiting for the stream)
1. Finally, you can go back to the main window and start streaming with the `Start Streaming` button.

Below you can see how to set the appropriate settings (steps 2) and 3) from the list of steps above):
![OBS settings](assets/OBS_settings.webp)

Once the streaming has started, you can visit `localhost:4000`, where the client application should be available. After a few seconds, you should be able to play
the video you are streaming. Most likely, a kind of latency will occur and the stream played by the client application will be delayed. That latency can reach values as great as 10 seconds, which is the result
of a relatively complex processing taking place in the pipeline. Note, that the latency here is not a result of the packets traveling far via the network, as the packets do not leave our computer in that setup - in the case of a real streaming server, we would need to take into consideration the time it takes a packet to reach the destination.


Running the RTMP to HLS demo

In this chapter, we will discuss the multimedia-specific part of the application.

## The pipeline

Let's start with `lib/rtmp_to_hls/pipeline.ex` file. All the logic is put inside the [`Membrane.Pipeline.handle_init/1`](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#c:handle_init/1) callback,
which is invoked once the pipeline is initialized.

**_`lib/rtmp_to_hls/pipeline.ex`_**

```elixir
 @impl true
 def handle_init(_opts) do
 ...
 children: %{
     src: %Membrane.RTMP.SourceBin{port: 9009},
     sink: %Membrane.HTTPAdaptiveStream.SinkBin{
         manifest_module: Membrane.HTTPAdaptiveStream.HLS,
         target_window_duration: 20 |> Membrane.Time.seconds(),
         muxer_segment_duration: 8 |> Membrane.Time.seconds(),
         storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{directory: "output"}
     }
 },
 ...
 end
```

First, we define the list of children. The following children are defined:

- `:src` - a `Membrane.RTMP.SourceBin`, an RTMP server, which, according to its `:port` configuration, will be listening on port `9009`. This bin will be acting as a source for our pipeline. For more information on RTMP Source Bin please visit [the documentation](https://hexdocs.pm/membrane_rtmp_plugin/Membrane.RTMP.SourceBin.html).
- `:sink` - a `Membrane.HTTPAdaptiveStream.SinkBin`, acting as a sink of the pipeline. The full documentation of that bin is available [here](https://hexdocs.pm/membrane_http_adaptive_stream_plugin/Membrane.HTTPAdaptiveStream.SinkBin.html). We need to specify some of its options:
- `:manifest_module` - a module which implements [`Membrane.HTTPAdaptiveStream.Manifest`](https://hexdocs.pm/membrane_http_adaptive_stream_plugin/Membrane.HTTPAdaptiveStream.Manifest.html#c:serialize/1) behavior. A manifest allows aggregate tracks (of a different type, i.e. an audio track and a video track as well as many tracks of the same type, i.e. a few video tracks with different resolutions). For each track, the manifest holds a reference to a list of segments, which form that track. Furthermore, the manifest module is equipped with the `serialize/1` method, which allows transforming that manifest to a string (which later on can be written to a file). In that case, we use a built-in implementation of a manifest module - the `Membrane.HTTPAdaptiveStream.HLS`, designed to serialize a manifest into a form required by HLS.
- `:target_window_duriation` - that value determines the minimal manifest's duration. The oldest segments of the tracks will be removed whenever possible if persisting them would result in exceeding the manifest duration.
- `:muxer_segment_duration` - the maximal duration of a segment. Each segment of each track shouldn't exceed that value. In our case, we have decided to limit the length of each segment to 8 seconds.
- `:storage` - the sink element, the module responsible for writing down the HLS playlist and manifest files. In our case, we use a pre-implemented `Membrane.HTTPAdaptiveStream.FileStorage` module, designed to write the files to the local filesystem. We configure it so that the directory where the files will be put in the `output/` directory (make sure that that directory exists as the storage module won't create it itself).

The fact that the configuration of a pipeline, which performs relatively complex processing, consists of just two elements, proves the power of [bins](/basic_pipeline_extension/02_Bin.md). Feel free to stop for a moment and read about them if you haven't done it yet.

After providing the children's specifications, we are ready to connect the pads between these children. Take a look at that part of the code:
**_`lib/rtmp_to_hls/pipeline.ex`_**

```elixir
 @impl true
 def handle_init(_opts) do
 ...
 links: [
     link(:src)
     |> via_out(:audio)
     |> via_in(Pad.ref(:input, :audio), options: [encoding: :AAC])
     |> to(:sink),
     link(:src)
     |> via_out(:video)
     |> via_in(Pad.ref(:input, :video), options: [encoding: :H264])
     |> to(:sink)
 ]
 ...
 end
```

The structure of links reflects the desired architecture of the application, described in the [chapter about system architecture](../videoroom/3_SystemArchitecture.md).
`:src` has two output pads: the `:audio` pad and the `:video` pad, transferring the appropriate media tracks.
The source's `:audio` pad is linked to the input `:audio` pad of the sink - along with the `:encoding` option. That option is an atom, describing the codec which is used to encode the media data - when it comes to audio data,
we will be using AAC coded.
At the time of the writing, only `:H264` and `:AAC` codecs are available to be passed as an `:encoding` option - the first one is used with video data, and the second one is used with audio data.
By analogy, the source's `:video` pad is linked with the sink's `:video` pad - and the `:encoding` to be used is H264.

The final thing that is done in the `handle_init/1` callback's implementation is returning the desired actions:
**_`lib/rtmp_to_hls/pipeline.ex`_**

```elixir
 @impl true
 def handle_init(_opts) do
 ...
 { {:ok, spec: spec, playback: :playing}, %{} }
 end
```

The first action is the `:spec` action, which spawns the children. The second action changes the playback state of the pipeline into the `:playing` - meaning, that data can start flowing through the pipeline.

## Starting the pipeline

The pipeline is started with `Supervisor.start_link`, as a child of the application, inside the `lib/rtmp_to_hls/application.ex` file:

**_`lib/rtmp_to_hls/application.ex`_**

```elixir
 @impl true
 def start(_type, _args) do
    children = [
        # Start the Pipeline
        Membrane.Demo.RtmpToHls,
        ...
    ]
    opts = [strategy: :one_for_one, name: RtmpToHls.Supervisor]
    Supervisor.start_link(children, opts)
 end
```

## HLS controller

The files produced with the pipeline are written down to the `output/` directory. We need to make them accessible via HTTP.
The Phoenix Framework provides tools to achieve that - take a look at the `RtmpToHlsWeb.Router`:
**_`lib/rtmp_to_hls_web/router.ex`_**

```elixir
scope "/", RtmpToHlsWeb do
    pipe_through :browser

    get "/", PageController, :index
    get "/video/:filename", HlsController, :index
 end
 ```

We are directing HTTP requests on `/video/:filename` to the HlsController, whose implementation is shown below:
**_`lib/rtmp_to_hls_web/controllers/hls_controller.ex`_**

```elixir
defmodule RtmpToHlsWeb.HlsController do
    use RtmpToHlsWeb, :controller

    alias Plug

    def index(conn, %{"filename" => filename}) do
        path = "output/#{filename}"

        if File.exists?(path) do
            conn |> Plug.Conn.send_file(200, path)
        else
            conn |> Plug.Conn.send_resp(404, "File not found")
        end
    end
end
```

This part of the code is responsible for sending the file `output/<filename>` to the HTTP request sender.


RTMP to HLS - pipeline

In this really short chapter, we will take a look at how one can play the video delivered with HLS on the website.

In our demo, we are taking advantage of the HLS.js player.
With such a tool onboard, the creation of the player is simple as that:
**_`lib/rtmp_to_hls_web/templates/page/index.html.heex`_**

```js
<script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>

<div class="container">
  <video id="player" controls autoplay class="Player"></video>
</div>
<script>
  var video = document.getElementById('player');
  var videoSrc = window.location.origin + `/video/index.m3u8`;
  if (Hls.isSupported()) {
    var hls = new Hls();
    hls.loadSource(videoSrc);
    hls.attachMedia(video);
  } else if (video.canPlayType('application/vnd.apple.mpegurl')) {
    video.src = videoSrc;
  }
</script>
```

First, we load a `hls.js` script.
Then, in the DOM we add a div holding a video element of a class `Player`.
Finally, we add our custom script, which decorates the video player element with the functionalities
provided by the `hls.js` library.
The `videoSrc` is a URL of the manifest file of the HLS playlist. As described in the previous chapter, the files from the playlist are available
at `/video/<filename>`.
If the HLS is supported by the client's browser, we create an object of type `Hls` (that class is a part of the `hls.js` library), then we set its source manifest file and a DOM element that acts as a video player. For more options that can be specified for the player, see the [documentation](https://github.com/video-dev/hls.js/blob/master/docs/API.md).


Web player

# Introduction

Other than RTMP, another protocol which is sometimes used for streaming media to the server is RTSP. It isn't so popular today anymore, however, there are still areas where it is still in use, primarily by surveillance cameras.
In the next couple chapters, we would like to show you a modification of the current RTMP to HLS converter, that will allow us to convert an RTSP stream instead.

## Use case

Suppose that we have a surveillance camera, which is providing us with an RTSP stream. We would like to stream this video to multiple viewers, for them to be easily accessible in the web browser.

Note, that we want this solution to be scalable as the number of users can be quite big.  

## Solution

The reasons for converting RTSP to HLS are very similar to the ones we explained with RTMP, as RTSP comes with similar shortcomings as RTMP. RTSP is very rarely supported on playback devices, has problems with traversing firewalls and proxies and doesn't support adaptive bitrate.

## What technology we will use

Before going further, make sure you have some basic understanding of the protocols used:
  
[Real Time Streaming Protocol (RTSP)](https://antmedia.io/rtsp-explained-what-is-rtsp-how-it-works) - what RTSP is and how does it work?

[Real-time Transport Protocol (RTP) and RTP Control Protocol (RTCP)](https://www.techtarget.com/searchnetworking/definition/Real-Time-Transport-Protocol) are protocols used for delivering audio and video over the internet. While the RTP transports the data, the RTCP is responsible for QoS and synchronization of multiple streams.

RTSP to HLS introduction

# Architecture

Now let's discuss how the architecture of our solution will look like.
It will be a little different from the RTMP to HLS architecture.
The main component will be the pipeline, which will ingest RTP stream and convert it to HLS. Beyond that we will also need a Connection Manager, which will be responsible for establishing an RTSP connection with the server.

![image](assets/rtsp_architecture.drawio.png)

When initializing, the pipeline will start a Connection Manager which starts an RTSP connection with the server. Once the connection is fully established, the pipeline will be notified.

Let's take a closer look on each of those components:

## Connection Manager
The role of the connection manager is to initialize RTSP session and start playing the stream.
It communicates with the server using the [RTSP requests](https://antmedia.io/rtsp-explained-what-is-rtsp-how-it-works/#RTSP_requests). In fact, we won't need many requests to start playing the stream - take a look at the desired message flow:

![image](assets/connection_manager.drawio.png)

First we want to get the details of the video we will be playing, by sending the `DESCRIBE` method. 
Then we call the `SETUP` method, defining the transport protocol (RTP) and client port used for receiving the stream.
Now we can start the stream using `PLAY` method.

## Pipeline

The pipeline consists of a couple elements, each of them performing a specific media processing task. You can definitely notice some similarities to the pipeline described in the [RTMP architecture](02_RTMP_Introduction.md). However, we will only be processing video so only the video processing elements will be necessary.

![image](assets/rtsp_pipeline.drawio.png)

We have already used the, `H264 Parser`, `MP4 H264 Payloader`, `CMAF Muxer` and `HLS Sink` elements in the RTMP pipeline, take a look at the [RTMP to HLS architecture](03_RTMP_SystemArchitecture.md) chapter for details of the purpose of those elements.

Let us describe briefly what is the purpose of the other components:

### UDP Source
This element is quite simple - it receives UDP packets from the network and sends their payloads to the next element.

### RTP SessionBin
RTP SessionBin is a Membrane's Bin, which is a Membrane's container used for creating reusable groups of elements. In our case the Bin handles the RTP session with the server, which has been set up by the Connection Manager.


RTSP to HLS system architecture

In the tutorial we won't explain how to implement the solution from the ground up - instead, we will run the existing code from [Membrane demos](https://github.com/membraneframework/membrane_demo).

To run the RTSP to HLS converter first clone the demos repo:
```console
git clone https://github.com/membraneframework/membrane_demo.git
```

```console
cd membrane_demo/rtsp_to_hls
```

Install the dependencies
```console
mix deps.get
```

Make sure you have those libraries installed as well:
- gcc
- libc-dev
- ffmpeg

On ubuntu:
```console
apt-get install gcc libc-dev ffmpeg
```

Take a look inside the `lib/application.ex` file. It's responsible for starting the pipeline.
We need to give a few arguments to the pipeline:
```elixir
@rtsp_stream_url "rtsp://rtsp.membrane.work:554/testsrc.264"
@output_path "hls_output"
@rtp_port 20000
```

The `@output_path` attribute defines the storage directory for hls files and the `@rtp_port` defines on which port we will be expecting the rtp stream, once the RTSP connection is established.

The `@rtsp_stream_url` attribute contains the address of the stream, which we will be converting. It is a sample stream prepared for the purpose of the demo. 

Now we can start the application:
```console
mix run --no-halt
```

The pipeline will start playing, after a couple of seconds the HLS files should appear in the `@output_path` directory. In order to play the stream we need to first serve them. We can do it using simple python server.

```console
python3 -m http.server 8000
```

Then we can play the stream using [ffmpeg](https://ffmpeg.org/), by pointing to the location of the manifest file:
```console
ffplay http://YOUR_MACHINE_IP:8000/rtsp_to_hls/hls_output/index.m3u8
```


Running the RTSP to HLS demo

Now let's focus on how the Connection Manager works. As mentioned previously its role is to establish the RTSP connection with the RTSP server.

The `ConnectionManager` module will use the [`Connection`](https://hexdocs.pm/connection/Connection.html) behaviour, which provides additional callbacks to [`GenServer`](https://hexdocs.pm/elixir/GenServer.html) behaviour, aiding with building a connection process.

First of all we are defining the `ConnectionStatus` struct, which we will use to keep the state of the ConnectionManager:

##### lib/connection_manager.ex
```elixir
defmodule ConnectionStatus do
  @moduledoc false
  @type t :: %__MODULE__{
          stream_url: binary(),
          rtsp_session: pid(),
          pipeline: pid(),
          keep_alive: pid(),
          pipeline_options: keyword()
          }

  @enforce_keys [
      :stream_url,
      :pipeline,
      :pipeline_options
  ]

  defstruct @enforce_keys ++
              [
                  :rtsp_session,
                  :keep_alive
              ]
end
```

It holds the `rtsp_session`, which is the pid of a process started with `Membrane.RTSP.start/1`. The `Membrane.RTSP` allows us to execute RTSP client commands. You can read more about it [here](https://hexdocs.pm/membrane_rtsp/readme.html).
The `pipeline` field is the pid of the pipeline, we will need it to notify the pipeline, that the RTSP connection is ready. In such notification we send `pipeline_options`, which contain necessary information about the stream.
The `keep_alive` is a process which repeatedly pings the RTSP server, in order to keep the connection alive and prevent a timeout.

Let's take a look at the `connect/2` callback, which is called immediately after the `init/1`:

##### lib/connection_manager.ex
```elixir
def connect(_info, %ConnectionStatus{} = connection_status) do
  rtsp_session = start_rtsp_session(connection_status)
  connection_status = %{connection_status | rtsp_session: rtsp_session}

  if is_nil(rtsp_session) do
    {:backoff, @delay, connection_status}
  else
    with {:ok, connection_status} <- get_rtsp_description(connection_status),
          :ok <- setup_rtsp_connection(connection_status),
          {:ok, connection_status} <- start_keep_alive(connection_status),
          :ok <- play(connection_status) do    

      send(
        connection_status.pipeline,
        {:rtsp_setup_complete, connection_status.pipeline_options}
      )

      {:ok, connection_status}
    else
      {:error, error_message} ->
        {:backoff, @delay, connection_status}
    end
  end
end
```

In the callback we go through the whole process of establishing RTSP connection - first starting the RTSP session, then getting the video parameters with, setting up the session, starting the keep alive process and finally playing the stream.
If all those steps succeed we can notify the pipeline, otherwise we back off and try to set up the connection after a `@delay` amount of time.

What might seem unclear to you is the `get_sps_pps` function.
It is responsible for getting the [sps and pps](https://www.cardinalpeak.com/blog/the-h-264-sequence-parameter-set) parameters from the RTSP DESCRIBE method. In short, sps and pps are parameters used by the H.264 codec and are required to decode the stream. Once the RTSP connection is complete we are sending them to the pipeline.

Connection manager

As explained in the [Architecture chapter](07_RTSP_Architecture.md), the pipeline will consist of a couple of elements, that will be processing the RTP stream.

The flow of the pipeline will consist of three steps. First, when the pipeline is initialized we will start the Connection Manager, which will set up the RTP stream via the RTSP.
Once that is finished, we will set up two initial elements in the pipeline - the `UDP Source` and `RTP SessionBin`, which will allow us to receive RTP packets and process them.
When the SessionBin detects that the RTP stream has been started, it will notify the pipeline with the `:new_rtp_stream` notification. Later on, we will add the remaining elements to the pipeline, allowing for the whole conversion process to take place.

Those steps take place, respectively, in the: `handle_init/1`, `handle_other/3` and `handle_notification/4` callbacks. While the `handle_init/1` is rather intuitive, we will describe in detail what's happening in the other callbacks.

Let us explain what's going on in the `handle_other` callback:

##### lib/pipeline.ex
```elixir
@impl true
def handle_other({:rtsp_setup_complete, options}, _ctx, state) do
  children = %{
    app_source: %Membrane.UDP.Source{
      local_port_no: state[:port],
      recv_buffer_size: 500_000
    },
    rtp: %Membrane.RTP.SessionBin{
      fmt_mapping: %{96 => {:H264, 90_000}}
    },
    hls: %Membrane.HTTPAdaptiveStream.Sink{
      manifest_module: Membrane.HTTPAdaptiveStream.HLS,
      target_window_duration: 120 |> Membrane.Time.seconds(),
      target_segment_duration: 4 |> Membrane.Time.seconds(),
      storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{
        directory: state[:output_path]
      }
    }
  }

  links = [
    link(:app_source)
    |> via_in(Pad.ref(:rtp_input, make_ref()))
    |> to(:rtp)
  ]

  spec = %ParentSpec{children: children, links: links}
  { {:ok, spec: spec}, %{state | video: %{sps: options[:sps], pps: options[:pps]}} }
end
```

When we receive the `rtsp_setup_complete` message, we first define the new children for the pipeline, and links between them - the UDP Source and the RTP SessionBin. We also create the HLS Sink, however we won't be linking it just yet. With the message we receive the sps and pps inside the options, and we add them to the pipeline's state.

Only after we receive the `:new_rtp_stream` notification we add the rest of the elements and link them with each other:

##### lib/pipeline.ex
```elixir
@impl true
def handle_notification({:new_rtp_stream, ssrc, 96, _extensions}, :rtp, _ctx, state) do
  actions =
    if Map.has_key?(state, :rtp_started) do
      []
    else
      children = %{
        video_nal_parser: %Membrane.H264.FFmpeg.Parser{
          sps: state.video.sps,
          pps: state.video.pps,
          skip_until_keyframe?: true,
          framerate: {30, 1},
          alignment: :au,
          attach_nalus?: true
        },
        video_payloader: Membrane.MP4.Payloader.H264,
        video_cmaf_muxer: Membrane.MP4.Muxer.CMAF
      }

      links = [
        link(:rtp)
        |> via_out(Pad.ref(:output, ssrc),
          options: [depayloader: Membrane.RTP.H264.Depayloader]
        )
        |> to(:video_nal_parser)
        |> to(:video_payloader)
        |> to(:video_cmaf_muxer)
        |> via_in(:input)
        |> to(:hls)
      ]

      [spec: %ParentSpec{children: children, links: links}]
    end

  { {:ok, actions}, Map.put(state, :rtp_started, true) }
end
```

First we check, if the stream hasn't started yet. That's because if we are restarting the pipeline there might be a previous RTP stream still being sent, so we might receive the `:new_rtp_stream` notification twice - once for the old and then for the new stream. We want to ignore any notification after the first one, as we want only a single copy of each media processing element.
Notice the sps and pps being passed to the H264 parser - they are necessary for decoding the stream.


RTSP to HLS - pipeline

As you can see converting an RTMP or RTSP stream to HLS might not be the easiest task. However, some understanding of the protocols and the usage of a couple of prefabricated elements in the Membrane Framework make up a rather compact solution. Importantly, such solution is open for modification and easily extensible.

HLS is one of the most popular media streaming protocols. Its advantages are using HTTP, allowing it for traversing over firewalls. It also allows for adaptive bitrate, i.e. it changes the resolution of the video based on the network bandwidth. Unlike RTMP or RTSP it is highly scalable. You will often come across HLS in the media streaming industry, so it's useful to have some understanding of it.

We hope that you grasped the basic ideas of the RTMP and RTSP protocols as well as that you know how to create your 'to HLS' converter with the use of the Membrane Framework.

If you liked this tutorial and want to extend your knowledge of the Membrane Framework, make sure to check out [other tutorials](https://membrane.stream/learn).


Summary

In this chapter you will learn about the H264 codec and how its processing is done with the Membrane Framework plugin - the [H264 plugin](https://github.com/membraneframework/membrane_h264_ffmpeg_plugin).

In H264 we can distinguish two layers of abstraction - the video coding layer (VCL), focused on the visual representation of the video, and the network abstraction layer (NAL), focused on how to structurize the stream so that it can be sent via the network.

When it comes to VCL, a video encoded with the H264 codec is represented as a sequence of pictures. Each picture consists of many **macroblocks**.
The macroblock is simply a part of the picture (in the context of some spatial dependence) - i.e. the top left corner of the picture.
The macroblocks are grouped together into so-called **slices**. Each slice consists of some macroblocks, which share the same **slice header**, containing some metadata common for all these macroblocks.

Each slice (which in fact is a part of a video picture), can be packed into a single
**NALu** (*Network Abstraction Layer units*). As the name suggests, here is where we start to deal with NAL. NALu is just an atomic piece of information sent through the network - it might contain some visual data (i.e. a list of macroblocks forming a part of a video frame), but it is not limited to the visual data - within NALus metadata used to properly decode the stream is also sent. We can distinguish two types of NAL units:

- VCL NALus - which stands for "Video coding layer" NALus
- Non-VCL NALus - which stands for "Non-video coding layer" NALus

There are different types of both VCL and Non-VCL units - for more information on them you can refer [here](https://yumichan.net/video-processing/video-compression/introduction-to-h264-nal-unit/)
> **Parse H264 stream on your own!**
> 
> With the use of Membrane Framework, you can inspect the types of NALus in your h264 file. To do so, you need to clone the H264 parser repository with:
>
> ```console
>   git clone https://github.com/membraneframework/membrane_h264_ffmepg_plugin
> ```
>
> Later on, inside the repository's directory, you can launch the Elixir's interactive shell, with compiled modules from the repository's mix project, by typing:
>
> ```console
>   iex -S mix
> ```
>
> Once in iex, you can do the following thing:
>
> ```elixir
>   alias Membrane.H264.FFmpeg.Parser.NALu
>
>   # Of course you can read your own file, here is just an example file from the membrane_rtmp_plugin's test directory,
>   # available at: https://github.com/membraneframework/membrane_rtmp_plugin/tree/master/test/fixtures/testvideo.h264
>   binaries = File.read!("test/fixtures/testvideo.h264") 
>
>   NALu.parse(binaries)
> ```
>
> You should see the following response:
>
> ```elixir
>   {[
>       %{
>           metadata: %{h264: %{new_access_unit: %{key_frame?: true}, type: :sps}},
>           prefixed_poslen: {0, 29},
>            unprefixed_poslen: {4, 25}
>        },
>        %{
>            metadata: %{h264: %{type: :pps}},
>            prefixed_poslen: {29, 9},
>            unprefixed_poslen: {33, 5}
>        },
>        %{
>            metadata: %{h264: %{type: :sei}},
>            prefixed_poslen: {38, 690},
>            unprefixed_poslen: {41, 687}
>         },
>        %{
>            metadata: %{h264: %{type: :idr}},
>            prefixed_poslen: {728, 8284},
>            unprefixed_poslen: {731, 8281}
>        },
>        %{
>            metadata: %{h264: %{type: :non_idr}},
>            prefixed_poslen: {9012, 1536},
>            unprefixed_poslen: {9016, 1532}
>        },
>        ....
>   ]}
> ```
>
> As you can see, NALus of different types have appeared, just like:
>
> - SPS - *Sequence Parameter Set*, a Non-VCL NALu, with a set of parameters that rarely change, applicable to the series of consecutive coded video pictures, called the **coded video sequence**. Each coded video sequence can be decoded independently of any other coded video sequence.
> - PPS - *Picture parameter Set* - a set of parameters that are applicable to some pictures from the coded video sequence. Furthermore, inside a PPS NALu there is a reference to SPS.
> - SEI - *Supplemental Enhancement Information* - some additional information that enhances the usability of the decoded video, i.e. timing information.
> - IDR - *Instantaneous Decoding Refresh*, a VCL unit containing the I-frame (known also as `intra frame`) - a picture that can be decoded without knowledge of any other frame, in contrast to P-frames and B-frames, which might need previously presented frames or frames that need to be presented in the future. As you might guess, the I-frames size is much greater than P-frame or B-frame size - that is because the whole information about the content of the picture needs to be encoded in such a frame.
> - NON_IDR - *Non-Instantaneous Decoding Refresh* - a VCL unit containing a P-frame or B-frame or parts of such a non-key frame. Note the size of a Non-IDR NALu (1536 B), compared to the size of IDR NALu (8284 B).

The sequence of NALus of a special form creates an **Access Unit**.
Each access unit holds a single picture of the video.
When such a picture is the keyframe, we refer to it as a **IDR Access Unit**. Otherwise, we call it **Non-IDR Access Unit**. Of course, the IDR Access Units are much bigger than Non-IDR Access Units when it comes to their binary size.
Sometimes, for the convenience of the decoding, the access units are separated with the use of another Non-VCL NALu, called *AUD* (**Access Unit Delimeter**).
Below you can find a diagram showing the structure of an access unit:
![Access Unit structure](assets/au_structure.png)
The access unit consists of the **primary coded picture** - a set of macroblocks representing the picture, and optionally of the **redundant coded picture** - the set of macroblocks that hold the redundant information about the given areas on the picture.
A parser needs to be aware that some of the NALus are optional are might not appear in the stream.

The existence of a coded video sequence is determined by the presence of an IDR NALu in the first access unit. Each coded video sequence can be decoded independently of the other coded video sequences.

As a summary we would like to present a diagram showing an exemplary NALus stream, along with the structures we can distinguish in that stream:
![H264 NALus stream](assets/h264_structure.png)


(Suplement) H264 codec

Membrane broadcasting tutorial

In this tutorial we will describe what is H264, why to use it and how to fetch interesting information from the H264 stream

# Introduction

## Once upon a time, there was a parser…

In Membrane, we have a concept of an H.264 parser - a processing element that is designed to put the incoming H.264 stream "in order" by splitting the stream into packets containing the given amount of information (i.e. into separate video frames) or fetching some information about the video (like resolution or framerate). Such processing is often required before muxing the stream into a container or putting it into some transport protocol's packets. 
We used to use an Elixir wrapper on ffmpeg SDK that performed the type of processing described above - we have called that wrapper H264.FFMpeg.Parser, and it's still available in our repository: membrane_h264_ffmpeg_plugin, along with the H264 software decoder and encoder. 
Initially, such a solution perfectly fit our needs - we had a concise yet bug-free parser implementation, and we didn't need to get our hands dirty with the h264 stream structure at all. The difficult stuff was already done by the great FFmpeg team, and we were just harvesting the fruits of their labor.

However, it wasn't that long until we faced a need to add some custom functionalities to the parser. Nothing particularly complicated: dropping frames from the stream before the parameters or keyframe appeared for the first time. 
That was the moment that we started struggling with a tremendous number of problems, which mostly originated in the fact that the state of the processing element was held in three different places:
* in the Elixir's GenServer state
* in the C bridge code between Elixir and FFMpeg SDK 
* internally, in the data structures provided by FFMpeg SDK
A stack of options, parameters and behavioral patterns had been growing, some options had become more or less explicitly incompatible with others, and more and more time was being spent on debugging the problems caused by the use of our parser. A decision was made - we need to perform a refactor. Soon we realized that there was in fact nothing to refactor - the whole code would have needed to be thrown out and the element would have needed to be written from scratch.

## The new hope

We realized that we were more experienced with multimedia processing than we had been when the wrapped parser was first created. So we asked ourselves - why not create the parser in plain Elixir and get rid of some native FFMpeg dependency? It shouldn't be that difficult, should it? …
Long story short, we learned a lot ;) And we would like to share some of our findings with you.


The first order of business was to put our knowledge in order and supplement it. There is no better knowledge source than the official ITU-T standard defining the H264 codec: "Advanced video coding for generic audiovisual services."
It might be overwhelming - the standard consists of 680 pages of printed text written in an ultra technical manner, however, for our needs, we could skip the parts concerning the video encoding and decoding processes - which make up most of the standard's volume.

## AVC, H.264 or MPEG-4 p. 10?
Firstly, we would like to explain the nomenclature that we will be using, since it might be misleading.
The codec, or more precisely - a coding standard, that we are describing is known under three names:
* **AVC**  - that stands for: Advanced Video Coding
* **H.264** - the name is derived from the name of **ITU-T** (*International Telecommunication Union  Standardization Sector*) (recommendation)[https://www.itu.int/rec/T-REC-H.264] that defines that coding standard 
* **MPEG-4 Part 10** - the name is derived from the name of a (document)[https://www.mpeg.org/standards/MPEG-4/10/] defining the standard, prepared by the **ISO/IEC JTC 1** (*Joint Technical Committee* (JTC) of the *International Organization for Standardization* (ISO) and the *International Electrotechnical Commission*) **Moving Picture Experts Group**

The existence of multiple names originates from the fact that the codec has been standardized as a cooperation of two groups - Video Coding Experts Group of ITU-T and Moving Picture Experts Group of ISO/IEC JTC 1. Within the text of the article we will most frequently use the "H.264" name, as it is most commonly used. 

## Why do we use H.264?
There are multiple reasons! The most important is that it allows us to represent video in a format that:
* supports compression - compression happens on multiple levels (for more information about some "tricks" used to compress the images and video, we invite you to read a tutorial published on [our website](https://membrane.stream/learn/digital_video_introduction)):
    * chroma subsampling - the compression standard, which takes advantage of the imperfections of human perception and removes the information about the  pixels' color that the human's eye can't even process
    * DCT (*discrete cosine transform*) and rounding it's coefficients to the integers
    * intra-frame prediction - each video frame is divided into so called "macroblocks" (square areas consisting of 16x16 pixels) and instead of storing the information about all the pixels of each frame, only information of all of the pixels for some frames is stored (these are so called "keyframes"), and the rest of the frames are stored as vectors describing the translation of given macroblocks
* allows for convenient multimedia transport - it's easy to pack an H.264 stream into multimedia containers (i.e. MP4) or packets (like RTP packets) and send it via the internet.
 
It's good to be aware that in fact the H.264 stream is designed to be wrapped in some other data structure, like a multimedia container (i.e. MP4) or some transport protocol packets (i.e. RTP packets) - we shouldn't store the H.264 stream as a plain binary file, as it might lead to many problems. The reason for that is that the H.264 stream itself does not contain all of the information that is required for a video to be properly displayed. Examples of such information are timestamps - H.264 stream does not contain timestamps at all, so unwrapping the H.264 stream from the MP4 container (that contains timing information additionally) and storing just the H.264 stream leads to a loss of information.

However, that's just theory - in practice, tools like FFmpeg allow you to read the plain H.264 files and do some "best effort" approximations of the missing information - i.e. when the framerate of the video is constant, they can simply calculate the missing timestamps by multiplying the ordering number of the given video frame by the inversion of the framerate.
That defines some kind of implicit "standard" of dealing with H.264 stream dumps, which we have also needed to take into consideration while working on our parser.

One might ask - why use H.264, when its successor, **H.265** standard (known also as **HEVC**, which stands for *high efficiency video compression*) is already available. The answer is - because H.264 is still widely used and still there is a need to cope with H.264 streams that are delivered to our system. At the same time, starting using H.265 comes with a great number of benefits, as it:
* allows for a better compression ratio. For a given video quality, up to 50% of bandwidth can be saved with H.265, compared to H.264, which has non-negligible impact on the maintenance costs of a media processing infrastructure.
* supports up to 8K UHDTV (in contrast to H.264, which supports up to 4K).
* supports up to 300 FPS (in contrast to H.264, which supports up to 59.94 FPS)

## H.264 layers - VCL vs NAL

In H.264 two so-called "layers" can be distinguished:
* **VCL** (*Video Coding Layer*) - this layer deals mostly with the video compression standard. H.264 encoders and decoders need to work with that layer, but in the case of the H.264 parser, we won't need to dig into it. 
* **NAL** (*Network Abstraction Layer*) - the layer that comes as a part of the network protocol's stack. It defines an abstraction over the VCL as it describes how the stream could be split into packets - Network Abstraction Layer units (we will refer to these units as NALu). Such an abstraction layer comes in handy for network devices i.e. it delivers information about how important a given packet is for the decoding process - based on that information, a network device knows which packets are high-priority and shouldn't be dropped (i.e. the ones that hold the keyframes), as dropping them would make the decoding impossible. 

The NAL was the subject of our interest when it comes to the H.264 parser - the parser's goal is to read the NAL units and fetch the information that interested us out of them.


![VCL vs NAL scheme](assets/images/VCL_vs_NAL.png)
<br>

What is H264

As noted above, from the NAL perspective, the H.264 stream is just a sequence of NAL units. 
But how do we know where one NAL unit ends and another one starts?
Well, the answer to that question is not straightforward, since the ITU specification does not define a single byte stream format. At the same time, one of the possible formats is described as an annex to the specification - Annex B. That format itself is usually referred to as Annex B and we will also stick to that name. 
We need to be aware that that format is not the only option - the other commonly used byte stream format is AVCC, known also as length prefix.
We will briefly discuss both of these formats.


## Annex B

Annex B is a format in which all NAL units are preceded with a so-called "start code" - one of the following byte sequences: three-byte `0x000001` or four-bytes `0x00000001`. 
In the case of that format, the H.264 stream might look as follows:
```
([0x000001][first NAL unit]) | ([0x000001][second NAL unit]) | ([0x000001][third NAL unit]) …
```
The existence of two "start codes" is dictated by the fact that the first variant is more applicable in certain situations than the second one. For a four-bytes variant, it's easy to find the start of a NAL unit sent through a serial connection, as it's easy to align the stream to 31 "0" bits followed by a single "1" bit. In other cases, to save space, it's common to use a three-byte variant.

The choice of `0x000001` and `0x00000001` as a NAL units separator wasn't unintentional - such a sequence of bytes does not encode much information (recall Shannon's formula for the amount of information) and that's why it's not frequent for the compression algorithm used in H.264 to produce such a sequence of bytes (the compression mechanism tries to encode as much information about video in the shortest bytes sequence possible). However, such a byte sequence might still appear in the stream (please also note, that some parts of the stream are metadata, which is not compressed) - and the format definition must take that into consideration. That's why an "emulation prevention" mechanism has been introduced. That mechanism is nothing more than a regular "escape character" mechanism - each of the following three-bytes sequences is not allowed to exist in the "escaped" byte stream:
* `0x000000` 
* `0x000001` 
* `0x000002`
* `0x000003`
and that's why it gets replaced correspondingly with the following sequences:
* `0x00000300`
* `0x00000301`
* `0x00000302`
* `0x00000303`
The "escape character" is 0x03 - also known as `emulation_prevention_three_byte`.

The representation with "emulation prevention three bytes" inserted is called **SODB** (*String of Data Bits*), and the representation without the "emulation prevention" bytes is called **RBSP** (*Raw Byte Sequence Payload*). Encoders need to take care of escaping the unallowed byte sequences (converting the byte stream from RBSP to SODB), while decoders are obliged with the reverse operation (converting from SODB to RBSP).


## AVCC

In this byte stream format, each NAL unit is preceded with a `nal_size` field, describing its length. That length might be stored with the use of 1, 2 or 4 bytes, and that is why a special additional header (known as "extradata", "sequence header" or "AVCDecoderConfigurationRecord") is required to be present in the stream to specify how many bytes are used to store the NAL unit's length. The stream then might look as follows: 
```
([extradata]) | ([length] NALu) | ([length] NALu) | …
```

The syntax of the extradata is described in MPEG-4 Part 15 "Advanced Video Coding (AVC) file format" section 5.2.4.1:
```
aligned(8) class AVCDecoderConfigurationRecord { 
    unsigned int(8) configurationVersion = 1; 
    unsigned int(8) AVCProfileIndication; 
    unsigned int(8) profile_compatibility; 
    unsigned int(8) AVCLevelIndication; 
    bit(6) reserved = '111111'b; 
    unsigned int(2) lengthSizeMinusOne; 
    bit(3) reserved = '111'b; 
    unsigned int(5) numOfSequenceParameterSets; 
    for (i=0; i< numOfSequenceParameterSets; i++) { 
        unsigned int(16) sequenceParameterSetLength;          
        bit(8*sequenceParameterSetLength) sequenceParameterSetNALUnit; 
    } 
    unsigned int(8) numOfPictureParameterSets; 
    for (i=0; i< numOfPictureParameterSets; i++) { 
        unsigned int(16) pictureParameterSetLength; 
        bit(8*pictureParameterSetLength) pictureParameterSetNALUnit; 
    }
}
```
The value of `lengthSizeMinusOne` field (which can be: 0, 1 or 3) determines how many bytes are used to store a length of NALu (that is, correspondingly, 1, 2 or 4 bytes). The most commonly used one is a 4-bytes size field that allows for NAL units of length up to `2^32-1` bytes.  

 
## NAL unit syntax

Each of the NAL units that we can find in the stream consists of a number of fields. Each field occupies a particular number of bytes and its value has a given semantics. ITU specifications define a syntax to describe NAL units' structure. We will become familiar with some field types defined within that syntax, as well see some exemplary NAL unit descriptions using that syntax.

### Data types

These are some of the types that can be met in the NAL units syntax (for all of the types available, please refer to chapter 7.2 of the ITU Specification):
* **f(N)** - "fixed pattern" sequence of N bits. I.e. f(8) might be some particular pattern of 8 bits - just like `00001111`. For information on how the pattern looks, you need to look in the semantics description of a given field. 
* **u(N)** - an unsigned integer written with the use of N bits - i.e. u(16) means an unsigned integer written with the use of 16 bits.
* **u(v)** - an unsigned integer written with the use of some number of bits. That number is usually specified by the value of some other, previously read field. In order to find out which field that is, you need to refer to the description of the semantics of that field.
* **ue(v)** - unsigned integer of variable size written with *Exponential-Golomb* coding. For more information about that coding you can refer [here](https://en.wikipedia.org/wiki/Exponential-Golomb_coding).
* **se(v)** - same as above, but this time the integer is signed.


### General NAL unit structure

Below there is a description of each NAL unit's structure. The structure description that you can find in chapter 7.3.1 of the ITU Specification is slightly different (and much more complicated). That is because it also covers the fact that there is a need to get rid of emulation_prevention_bytes. In order to simplify things, we assume that in the structure description we use, the byte stream has already been converted to RBSP. 

| nal_unit( NumBytesInNALunit )                          | Descriptor |
|------------------------------------------------------- |------------|
|{                                                       |            |
|   &nbsp;&nbsp;&nbsp;&nbsp; forbidden_zero_bit          |f(1)        |
|   &nbsp;&nbsp;&nbsp;&nbsp; nal_ref_idc                 |u(2)        |
|   &nbsp;&nbsp;&nbsp;&nbsp; nal_unit_type               |u(5)        |
|   &nbsp;&nbsp;&nbsp;&nbsp; nal_unit_data()             |            |
|}                                                       |            |

As you can see, each NAL unit starts with a single byte being a kind of a NAL unit header. Its first bit (`forbidden_zero_bit`) is always equal to 0, then comes a `nal_ref_idc` field consisting of two bits, and finally `nal_unit_type` - an unsigned integer written with the use of 5 bits, which determines the type of NAL unit. 

## NALu types

Each NAL unit can be of a different type. In this chapter we will describe only some of the NAL units types, while all the possible types are listed in the table available in "Appendix/NALu types".

There are two "classes" of NAL units types defined in ITU-T Specification's Annex A - **VCL** and **non-VCL** NAL units. The first one holds the encoded video, while the other does not contain video data at all. In our case, we found the following NAL unit types especially helpful:
* VCL:
    * *Coded slice of a non-IDR picture* (**non-IDR**) - contains a part or a complete non-keyframe (that is: P-frame or a B-frame)
    * *Coded slice of an IDR picture* (**IDR**) -  contains a part or a complete keyframe (also known as I-frame). The name IDR (that stands for *instantaneous decoding refresh*) originates from the fact that the decoder can "forget" the previous frames when the new keyframe appears, since it contains the complete information about the frame. 
* non-VCL:
    * *Sequence parameter set* (**SPS**) - contains metadata that is applicable to one or more *coded video sequences*. In that NALu you will find information allowing you to calculate the video resolution or H.264 profile.
    * *Picture parameter set* (**PPS**) - contains metadata applicable to one or more *coded pictures* 
    * *Access unit delimiter* (**AUD**) - just a separator between *access units*
    * *Supplemental enhancement information* (**SEI**) - contains some additional metadata that "assist in processes related to decoding, display or other purposes". At the same time, information stored in SEI is not required to restore the picture during the decoding process, so the decoders are not obliged to process SEI. In fact, SEI is defined as Annex D to the ITU specification. 

Some of the terms used in these (especially the ones written with *curved* font) might not be known to you yet - don't worry, keep on reading as they will be explained later. Once that happens, we invite you to go back to that section to reread it ;)


H264 NAL units

A video is a sequence of *frames* - images. Each frame might be (but it's not obliged to be!) written with [interlacing](https://en.wikipedia.org/wiki/Interlaced_video) - which means: it might consist of two *fields*. In such a case, a *field* is a group of alternating rows of a frame (even or odd ones). The two fields that form a single frame are referred to as the *bottom field* and the *top field*. 

In H.264, the term *picture* refers to both a single field of a frame, or to the frame itself. 

As noted previously, a H.264 stream consists of NAL units. Some NAL units hold the compressed video, while the others hold some metadata. The NAL units that hold video contain it in the form of a *slice*. Each slice is a group of *macroblocks* (along with a so-called *slice header*, containing some metadata). Right now it's sufficient for you to think about a macroblock as an area of 16x16 pixels on an image. Each picture is partitioned into macroblocks and such a division allows intra-frame prediction, as the translation vectors are defined between macroblocks of two pictures. We have reached the bottom - a macroblock is an atomic term in the context of H.264. 

Getting back to the slices - one or more slices create a *coded picture* - a compressed representation of a picture or a part of a picture (remember that a picture is a single video frame or a field). A coded picture might be one of two types:
* a *primary coded picture* -  a coded representation of a whole picture - it contains all the macroblocks of a picture (that means, that it's enough for a decoder to be able to display a picture based just on the primary coded picture)
* a *redundant coded picture* - a coded representation of a part of or a whole picture - as the name suggests ("redundant"), it repeats some information contained by a given primary coded picture
With the primary coded picture defined, we can define another crucial term used in H.264 - *access unit*.
An access unit is a sequence of NAL units that contain exactly one primary coded picture. Apart from the primary coded picture, a single access unit might also contain some *redundant coded pictures* (as well as some other video data or metadata, but let's not dig into the details). The crucial thing is that decoding a single access unit allows one to obtain a single *decoded picture*.
Below there is a scheme that shows a possible structure of an access unit:
![AU Scheme](assets/images/AU_scheme.png)
<br>     

As shown, an access unit might or might not start with a special type of NALu - access unit delimiter. Then comes zero or more SEI NAL units. Finally, there is the only obligatory part of the access unit - the sequence of NAL units that form a primary coded picture. Then comes zero or more sequences of NAL units forming redundant coded pictures. Then there is optionally the end of sequence NAL unit (it might help in separating the coded video sequences - you will learn what a coded video sequence is very soon). Finally, the end of stream NAL unit might optionally be present if the stream ends (but in the same manner as previously, it's not obligatory - the stream might end without the End of stream NAL unit appearing). 
The access unit might be of two types:
* IDR access unit - an access unit, in which primary coded picture is an IDR picture
* non-IDR access unit - an access unit, in which primary coded picture is not an IDR picture
Finally, we can define a coded video sequence - a sequence of access units that consists of exactly one IDR access unit at the beginning of the sequence, followed by zero or more non-IDR access units.

## Example - H.264 stream structure

Below there is an exemplary H.264 stream consisting of multiple NAL units. The NAL units are forming Access Units, and the Access Units are gathered together in the form of a Coded Video Sequence.
![H264 Stream Example](assets/images/H264_example_stream.png)
<br>     

Let's thoroughly inspect that stream's structure. 
First, there is a Sequence Parameter Set NAL. It contains some metadata common for zero or more coded video sequences. 
It's followed by two Picture Parameter Set NAL units - each of them contains some metadata relevant for zero or more coded pictures. In the slice's header of each NALu holding the video data (that is - IDR NALu or non-IDR NALu), there is a reference to the particular Picture Parameter Set NALu. Note that all the slices forming a single coded picture must refer to the same Picture Parameter Set NALu. In each Picture Parameter Set NALu, there is a reference to a Sequence Parameter Set - all the Picture Parameter Sets referred to by coded pictures from the same coded video sequence must refer to the same Sequence Parameter Set.
Right after the second Picture Parameter Set NALu a coded video sequence starts. It consists of three access units - the first one is an IDR access unit, while the remaining two are non-IDR access units. 
The first access unit starts with an optional SEI NALu, followed by two IDR NAL units. These two IDR NAL units form a single primary coded picture. The second access unit consists of two non-IDR NAL units, which also form a single primary coded video. That second access unit is probably much smaller in size compared to the first access unit - that is because it doesn't contain a keyframe. The last access unit starts with an optional SEI NALu, followed by a non-IDR NALu (which holds a complete primary coded picture) and an optional redundant coded picture that is contained in a single non-IDR NALu.


H264 stream structure

Fetching information from stream

| nal_unit_type | Content of NAL unit & RBSP syntax structure                                                              | NAL unit type class [Annex A] |
| ------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------ |
| 0             | Unspecified                                                                                              | non-VCL                        |
| 1             | Coded slice of a non-IDR picture *slice_layer_without_partitioning_rbsp()*                               | VCL                            |
| 2             | Coded slice data partition A *slice_data_partition_a_layer_rbsp()*                                       | VCL                            |
| 3             | Coded slice data partition B *slice_data_partition_b_layer_rbsp()*                                       | VCL                            |
| 4             | Coded slice data partition C *slice_data_partition_c_layer_rbsp()*                                       | VCL                            |
| 5             | Coded slice of an IDR picture *slice_layer_without_partitioning_rbsp()*                                  | VCL                            |
| 6             | Supplemental enhancement information (SEI) *sei_rbsp()*                                                  | non-VCL                        |
| 7             | Sequence parameter set *seq_parameter_set_rbsp()*                                                        | non-VCL                        |
| 8             | Picture parameter set *pic_parameter_set_rbsp()*                                                         | non-VCL                        |
| 9             | Access unit delimiter *access_unit_delimiter_rbsp()*                                                     | non-VCL                        |
| 10            | End of sequence *end_of_seq_rbsp()*                                                                      | non-VCL                        |
| 11            | End of stream *end_of_stream_rbsp()*                                                                     | non-VCL                        |
| 12            | Filler data *filler_data_rbsp()*                                                                         | non-VCL                        |
| 13            | Sequence parameter set extension *seq_parameter_set_extension_rbsp()*                                    | non-VCL                        |
| 14            | Prefix NAL unit *prefix_nal_unit_rbsp()*                                                                 | non-VCL                        |
| 15            | Subset sequence parameter set *subset_seq_parameter_set_rbsp()*                                          | non-VCL                        |
| 16 - 18       | Reserved                                                                                                 | non-VCL                        |
| 19            | Coded slice of an auxiliary coded picture without partitioning *slice_layer_without_partitioning_rbsp()* | non-VCL                        |
| 20            | Coded slice extension *slice_layer_extension_rbsp()*                                                     | non-VCL                        |
| 21            | Coded slice extension for depth view components *slice_layer_extension_rbsp()* (specified in Annex I)    | non-VCL                        |
| 22 - 23       | Reserved                                                                                                 | non-VCL                        |
| 24 - 31       | Unspecified                                                                                              | non-VCL                        |

Start your journey with Multimedia

Meetings & workshops